Jul 28 00:44:33.370380 ip-10-0-140-154 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 00:44:33.370391 ip-10-0-140-154 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 00:44:33.370401 ip-10-0-140-154 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 00:44:33.370712 ip-10-0-140-154 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 00:44:43.495342 ip-10-0-140-154 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 00:44:43.495358 ip-10-0-140-154 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 466d6f79e96144d78ac083458f85bf1b --
Jul 28 00:46:35.145533 ip-10-0-140-154 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 00:46:35.494893 ip-10-0-140-154 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 00:46:35.494893 ip-10-0-140-154 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 00:46:35.494893 ip-10-0-140-154 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 00:46:35.494893 ip-10-0-140-154 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 00:46:35.494893 ip-10-0-140-154 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 00:46:35.495847 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.495780    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 00:46:35.503231 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503199    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:35.503231 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503214    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:35.503231 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503217    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:35.503231 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503235    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:35.503231 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503239    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503242    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503245    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503247    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503250    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503253    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503255    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503259    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503263    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503267    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503270    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503273    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503275    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503278    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503281    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503283    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503286    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503288    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503291    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503294    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:35.503418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503296    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503299    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503303    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503305    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503308    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503310    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503314    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503316    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503322    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503325    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503328    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503331    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503334    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503337    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503340    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503343    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503346    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503349    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503351    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:35.503890 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503355    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503357    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503360    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503362    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503365    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503367    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503370    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503373    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503375    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503378    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503381    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503385    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503389    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503392    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503395    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503398    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503400    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503405    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503408    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503411    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:35.504418 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503414    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503417    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503420    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503423    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503426    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503430    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503434    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503439    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503442    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503445    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503448    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503450    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503454    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503457    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503460    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503463    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503465    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503468    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503471    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:35.504906 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503475    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503478    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503481    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503483    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503487    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503489    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503492    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503494    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503497    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503499    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503502    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503504    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503507    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503510    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503512    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503515    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.503517    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504186    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504192    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504195    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:35.505459 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504198    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504201    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504204    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504207    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504209    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504212    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504215    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504217    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504240    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504244    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504247    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504251    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504253    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504256    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504260    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504262    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504265    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504269    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504273    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:35.505941 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504276    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504280    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504284    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504286    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504289    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504292    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504296    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504298    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504301    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504304    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504307    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504309    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504312    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504316    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504319    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504322    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504325    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504329    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504331    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:35.506425 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504334    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504337    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504339    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504342    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504344    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504347    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504349    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504352    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504354    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504357    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504360    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504362    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504365    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504367    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504370    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504373    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504375    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504377    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504380    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504383    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:35.506882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504386    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504389    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504392    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504394    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504397    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504399    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504402    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504405    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504408    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504412    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504415    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504418    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504420    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504429    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504433    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504436    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504439    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504441    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504444    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504446    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:35.507378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504449    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504451    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504455    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504458    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504461    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504463    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504466    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504468    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504471    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504473    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504476    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504479    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504482    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504484    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504487    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504490    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504492    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504495    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504857    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504861    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:35.507857 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504865    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504867    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504870    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504873    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504875    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504878    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504880    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504883    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504886    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504890    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504894    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504898    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504901    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504904    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504908    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504911    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504914    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504917    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504919    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:35.508378 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504922    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504924    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504927    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504930    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504932    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504935    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504937    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504940    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504942    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504946    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504948    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504951    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504953    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504956    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504958    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504961    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504963    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504965    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504968    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:35.508848 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504970    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504973    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504975    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504978    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504980    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504983    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504985    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504988    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504991    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504994    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.504997    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505000    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505002    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505004    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505007    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505009    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505012    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505014    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505016    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505019    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:35.509368 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505021    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505023    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505026    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505028    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505031    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505033    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505036    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505039    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505041    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505043    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505046    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505048    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505051    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505053    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505056    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505058    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505061    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505063    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505066    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:35.509855 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505068    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505071    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505073    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505077    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505080    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505082    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505085    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505087    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505090    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505092    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505095    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505099    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505102    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505105    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505107    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505110    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505112    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505118    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505121    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505123    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:35.510333 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505192    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505198    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505205    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505217    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505233    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505236    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505240    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505245    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505248    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505251    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505254    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505257    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505260    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505263    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505266    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505269    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505273    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505276    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505281    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505284    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505287    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505290    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505293    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505297    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 00:46:35.510819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505300    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505303    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505306    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505309    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505312    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505317    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505320    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505323    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505327    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505330    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505333    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505336    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505338    2576 flags.go:64] FLAG: --enable-server="true"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505341    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505345    2576 flags.go:64] FLAG: --event-burst="100"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505348    2576 flags.go:64] FLAG: --event-qps="50"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505350    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505354    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505356    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505360    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505363    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505366    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505369    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505372    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505375    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 00:46:35.511400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505378    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505380    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505383    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505386    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505389    2576 flags.go:64] FLAG: --feature-gates=""
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505392    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505395    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505398    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505401    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505404    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505407    2576 flags.go:64] FLAG: --help="false"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505410    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505415    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505417    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505420    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505423    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505428    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505431    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505434    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505436    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505439    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505442    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505445    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505448    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 28 00:46:35.512062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505451    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505454    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505457    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505459    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505462    2576 flags.go:64] FLAG: --lock-file=""
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505465    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505468    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505471    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505477    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505480    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505483    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505486    2576 flags.go:64] FLAG: --logging-format="text"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505489    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505492    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505495    2576 flags.go:64] FLAG: --manifest-url=""
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505498    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505502    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505505    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505508    2576 flags.go:64] FLAG: --max-pods="110"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505511    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505515    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505518    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505521    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505523    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505528    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 00:46:35.512725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505531    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505539    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505542    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505545    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505548    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505550    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505556    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505559    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505562    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505579    2576 flags.go:64] FLAG: --port="10250"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505583    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505587    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e73eaa02e7ce047f"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505590    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505593    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505596    2576 flags.go:64] FLAG: --register-node="true"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505599    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505602    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505606    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505609    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505612    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505615    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505618    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505621    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505624    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505627    2576 flags.go:64] FLAG: --runonce="false"
Jul 28 00:46:35.513413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505630    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505633    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505636    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505639    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505642    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505645    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505650    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505653    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505655    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505658    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505661    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505664    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505667    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505670    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505673    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505678    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505681    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505683    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505687    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505690    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505693    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505696    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505699    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505702    2576 flags.go:64] FLAG: --v="2"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505705    2576 flags.go:64] FLAG: --version="false"
Jul 28 00:46:35.514082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505710    2576 flags.go:64] FLAG: --vmodule=""
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505713    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.505717    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505831    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505835    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505838    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505841    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505844    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505846    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505849    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505853    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505855    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505858    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505861    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505864    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505868    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505872    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505875    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505878    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:35.514811 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505880    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505884    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505887    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505890    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505892    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505895    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505898    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505900    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505903    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505906    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505908    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505911    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505913    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505916    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505919    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505922    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505924    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505927    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505929    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:35.515298 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505932    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505935    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505937    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505940    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505943    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505946    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505949    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505952    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505955    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505958    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505960    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505963    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505966    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505968    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505971    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505973    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505976    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505979    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505981    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505984    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:35.515766 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505986    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505989    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505992    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505994    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505997    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.505999    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506002    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506005    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506008    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506010    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506013    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506015    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506018    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506020    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506023    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506025    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506030    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506032    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506035    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:35.516261 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506039    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506041    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506044    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506047    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506049    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506052    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506054    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506057    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506060    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506062    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506065    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506068    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506070    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506073    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506075    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506078    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506080    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506083    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506085    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506088    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:35.516722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506091    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506095    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506099    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506102    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.506105    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.506669    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.513141    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.513156    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.513982    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.513998    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514086    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514093    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514098    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514103    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514108    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:35.517253 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514113    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514118    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514123    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514127    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514132    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514136    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514141    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514145    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514149    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514155    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514161    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514166    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514172    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514177    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514182    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514186    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514190    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514194    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514198    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:35.517644 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514202    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514207    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514212    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514217    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514235    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514240    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514245    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514249    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514253    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514256    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514259    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514261    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514264    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514267    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514269    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514272    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514275    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514277    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514280    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514282    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:35.518089 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514285    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514287    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514290    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514292    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514295    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514299    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514303    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514306    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514309    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514311    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514314    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514316    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514319    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514321    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514324    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514326    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514328    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514331    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514335    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514338    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:35.518746 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514341    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514343    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514346    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514349    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514351    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514354    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514356    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514358    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514361    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514364    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514367    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514369    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514371    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514374    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514376    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514379    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514381    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514384    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514386    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514389    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:35.519616 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514391    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514394    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514397    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514399    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514402    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514404    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514406    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514409    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514411    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514414    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514418    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514423    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514426    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514429    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:35.520122 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514432    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.514437    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514547    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514552    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514556    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514559    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514562    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514564    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514567    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514569    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514572    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514574    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514577    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514579    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514582    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 00:46:35.520550 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514585    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514587    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514590    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514592    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514595    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514598    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514603    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514606    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514609    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514611    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514615    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514618    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514621    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514624    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514628    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514639    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514642    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514644    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514647    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 00:46:35.520993 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514650    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514653    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514656    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514666    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514669    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514671    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514674    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514676    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514679    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514681    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514684    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514686    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514688    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514691    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514693    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514697    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514700    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514703    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514706    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 00:46:35.521560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514709    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514712    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514715    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514718    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514721    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514723    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514726    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514728    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514731    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514734    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514742    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514744    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514747    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514749    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514752    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514754    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514757    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514759    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514762    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514764    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 00:46:35.522189 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514767    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514769    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514772    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514774    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514776    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514779    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514781    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514784    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514786    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514788    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514791    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514793    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514796    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514798    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514800    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514803    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514805    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514808    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514810    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514813    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 00:46:35.523042 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514818    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514821    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514823    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514827    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514829    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514832    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514834    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:35.514837    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.514842    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.514944    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.517073    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.518373    2576 server.go:1019] "Starting client certificate rotation"
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.518490    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 00:46:35.523640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.518543    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 00:46:35.523961 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.520805    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 00:46:35.526102 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.526080    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 00:46:35.537358 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.537342    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 28 00:46:35.539533 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.539519    2576 log.go:25] "Validated CRI v1 image API"
Jul 28 00:46:35.540374 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.540349    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 00:46:35.543144 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.543125    2576 fs.go:135] Filesystem UUIDs: map[3990aec5-ef36-431d-a3cc-84795a215623:/dev/nvme0n1p4 4301cbe1-e930-46f9-a41b-fc2a0e57db89:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 00:46:35.543216 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.543143    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 00:46:35.545720 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.545698    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 00:46:35.548296 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.548172    2576 manager.go:216] Machine: {Timestamp:2026-07-28 00:46:35.546971085 +0000 UTC m=+0.296109800 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102586 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2aab3f814c01aa0d4d1b13de1c10a9 SystemUUID:ec2aab3f-814c-01aa-0d4d-1b13de1c10a9 BootID:466d6f79-e961-44d7-8ac0-83458f85bf1b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c9:e7:28:36:07 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c9:e7:28:36:07 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:b1:68:5b:b8:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 00:46:35.548296 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.548286    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 00:46:35.548449 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.548354    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 00:46:35.551417 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.551395    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 00:46:35.551569 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.551421    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-154.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 00:46:35.552757 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.552739    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 00:46:35.552825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.552764    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 00:46:35.552825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.552780    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 00:46:35.553443 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.553431    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 00:46:35.554407 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.554396    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 00:46:35.557080 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557065    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 00:46:35.557240 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557215    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 00:46:35.557298 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557266    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 00:46:35.557298 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557280    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 00:46:35.557393 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557300    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 00:46:35.557393 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557313    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 28 00:46:35.557393 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557336    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 00:46:35.557666 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557652    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 00:46:35.557731 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.557675    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 00:46:35.559762 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.559745    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 00:46:35.561691 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.561677    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 00:46:35.561756 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.561706    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 00:46:35.563475 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563463    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 00:46:35.563475 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563478    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563484    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563490    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563499    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563504    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563512    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563517    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563524    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563529    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563538    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 00:46:35.563549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.563543    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 00:46:35.564279 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.564266    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 00:46:35.564279 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.564282    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 00:46:35.567057 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.567043    2576 server.go:1268] "Started kubelet"
Jul 28 00:46:35.567336 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.567313    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 00:46:35.567393 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.567353    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 00:46:35.567786 ip-10-0-140-154 systemd[1]: Started Kubernetes Kubelet.
Jul 28 00:46:35.569693 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.567521    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 00:46:35.570136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.570114    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 00:46:35.570729 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.570706    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-154.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 00:46:35.571032 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.571003    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xzj4s"
Jul 28 00:46:35.571117 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.571066    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 00:46:35.571166 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.571115    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 00:46:35.573116 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.573098    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 00:46:35.574622 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.574603    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 00:46:35.576253 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.575430    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-154.ec2.internal.18c64d92e2f174c2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-154.ec2.internal,UID:ip-10-0-140-154.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-154.ec2.internal,},FirstTimestamp:2026-07-28 00:46:35.567019202 +0000 UTC m=+0.316157918,LastTimestamp:2026-07-28 00:46:35.567019202 +0000 UTC m=+0.316157918,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-154.ec2.internal,}"
Jul 28 00:46:35.577885 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.577869    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xzj4s"
Jul 28 00:46:35.582935 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.582919    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 00:46:35.583020 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.582934    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 00:46:35.583080 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.583044    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:35.583688 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.583668    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 00:46:35.583688 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.583668    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 00:46:35.583825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.583698    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 00:46:35.583825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.583785    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 00:46:35.583825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.583794    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 00:46:35.584104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584012    2576 factory.go:55] Registering systemd factory
Jul 28 00:46:35.584104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584029    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 28 00:46:35.584336 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584320    2576 factory.go:153] Registering CRI-O factory
Jul 28 00:46:35.584336 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584336    2576 factory.go:223] Registration of the crio container factory successfully
Jul 28 00:46:35.584490 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584397    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 00:46:35.584490 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584419    2576 factory.go:103] Registering Raw factory
Jul 28 00:46:35.584490 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584443    2576 manager.go:1196] Started watching for new ooms in manager
Jul 28 00:46:35.584862 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.584844    2576 manager.go:349] Starting recovery of all containers
Jul 28 00:46:35.586076 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.586022    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-154.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 00:46:35.586144 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.586122    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 00:46:35.595483 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.595468    2576 manager.go:354] Recovery completed
Jul 28 00:46:35.601303 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.601290    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.603181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.603166    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.603269 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.603196    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.603269 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.603233    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.604339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.604322    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 00:46:35.604339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.604338    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 00:46:35.604487 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.604355    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 00:46:35.604487 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.604392    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.606058 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.606042    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.606141 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.606068    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.606141 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.606078    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.607369 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.607356    2576 policy_none.go:49] "None policy: Start"
Jul 28 00:46:35.607405 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.607373    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 00:46:35.607405 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.607383    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 00:46:35.609383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.609371    2576 policy_none.go:47] "Start"
Jul 28 00:46:35.642883 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.642787    2576 manager.go:337] "Starting Device Plugin manager"
Jul 28 00:46:35.642967 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.642896    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 00:46:35.642967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.642906    2576 server.go:87] "Starting device plugin registration server"
Jul 28 00:46:35.643117 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.643107    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 00:46:35.643147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.643119    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 00:46:35.643249 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.643217    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 00:46:35.643343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.643322    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 00:46:35.643343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.643333    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 00:46:35.643768 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.643747    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 00:46:35.643842 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.643800    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:35.742903 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.742868    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.743239    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744099    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744105    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744128    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744136    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744140    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744156    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.744168    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.745147 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.744192    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 00:46:35.746466 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.746449    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:35.749902 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.749882    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.749974 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.749903    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-154.ec2.internal\": node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:35.774909 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.774895    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:35.845151 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.845126    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"]
Jul 28 00:46:35.845272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.845198    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.848558 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.848539    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.848644 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.848570    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.848644 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.848585    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.850891 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.850880    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.850994 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.850980    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.851039 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851017    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.851660 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851644    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.851748 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851675    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.851748 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851690    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.851748 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851642    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.851748 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851739    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.851748 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.851749    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.853959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.853943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.854037 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.853970    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 00:46:35.856192 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.856174    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 00:46:35.856285 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.856206    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 00:46:35.856285 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.856247    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeHasSufficientPID"
Jul 28 00:46:35.872827 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.872805    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-154.ec2.internal\" not found" node="ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.875433 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.875417    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:35.878018 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.878003    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-154.ec2.internal\" not found" node="ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.885697 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.885679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49d5be5d46952d796bae73c6a91a14e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"49d5be5d46952d796bae73c6a91a14e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.885783 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.885709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d5be5d46952d796bae73c6a91a14e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"49d5be5d46952d796bae73c6a91a14e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.885783 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.885739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/851ad20362c20c22501be689ae7a68aa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-154.ec2.internal\" (UID: \"851ad20362c20c22501be689ae7a68aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.976193 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:35.976170    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:35.986575 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.986555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49d5be5d46952d796bae73c6a91a14e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"49d5be5d46952d796bae73c6a91a14e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.986653 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.986615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/49d5be5d46952d796bae73c6a91a14e0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"49d5be5d46952d796bae73c6a91a14e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.986653 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.986646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d5be5d46952d796bae73c6a91a14e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"49d5be5d46952d796bae73c6a91a14e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.986741 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.986670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/851ad20362c20c22501be689ae7a68aa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-154.ec2.internal\" (UID: \"851ad20362c20c22501be689ae7a68aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.986741 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.986702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/851ad20362c20c22501be689ae7a68aa-config\") pod \"kube-apiserver-proxy-ip-10-0-140-154.ec2.internal\" (UID: \"851ad20362c20c22501be689ae7a68aa\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:35.986804 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:35.986736    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d5be5d46952d796bae73c6a91a14e0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal\" (UID: \"49d5be5d46952d796bae73c6a91a14e0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:36.077031 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.076996    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.182573 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.182533    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.182708 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.182589    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:36.182708 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.182640    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:36.286273 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.286191    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.390553 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.390527    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.491490 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.491449    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.500430 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.500409    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:36.517599 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.517579    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 00:46:36.517705 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.517688    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 00:46:36.517749 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.517728    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 00:46:36.584542 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.584512    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 00:41:36 +0000 UTC" deadline="2028-04-14 01:54:25.18791158 +0000 UTC"
Jul 28 00:46:36.584542 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.584535    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15025h7m48.603379338s"
Jul 28 00:46:36.592580 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.592561    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.692906 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.692878    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.792994 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.792958    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.804808 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:36.804776    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d5be5d46952d796bae73c6a91a14e0.slice/crio-44346a4ae7855c9bf5c3a37a721d58093b4330d6aeba6751e67b44fead687649 WatchSource:0}: Error finding container 44346a4ae7855c9bf5c3a37a721d58093b4330d6aeba6751e67b44fead687649: Status 404 returned error can't find the container with id 44346a4ae7855c9bf5c3a37a721d58093b4330d6aeba6751e67b44fead687649
Jul 28 00:46:36.810609 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:36.810590    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 00:46:36.816451 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:36.816429    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851ad20362c20c22501be689ae7a68aa.slice/crio-716a699d2e0461af373faf4684934bb0a4c4256307f2886eb4e7f62d0c5b8724 WatchSource:0}: Error finding container 716a699d2e0461af373faf4684934bb0a4c4256307f2886eb4e7f62d0c5b8724: Status 404 returned error can't find the container with id 716a699d2e0461af373faf4684934bb0a4c4256307f2886eb4e7f62d0c5b8724
Jul 28 00:46:36.893500 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.893445    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:36.994599 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:36.994574    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-154.ec2.internal\" not found"
Jul 28 00:46:37.038136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.038119    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:37.089824 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.089802    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:37.183386 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.183337    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:37.195366 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.195344    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 00:46:37.196309 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.196289    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal"
Jul 28 00:46:37.204999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.204981    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 00:46:37.558617 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.558584    2576 apiserver.go:52] "Watching apiserver"
Jul 28 00:46:37.567473 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.567268    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 00:46:37.567756 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.567724    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-2575g","openshift-multus/multus-2fvjp","openshift-network-operator/iptables-alerter-rsltp","openshift-ovn-kubernetes/ovnkube-node-j849f","kube-system/global-pull-secret-syncer-zw59q","kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq","openshift-dns/node-resolver-z427m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal","openshift-multus/multus-additional-cni-plugins-frfxj","openshift-multus/network-metrics-daemon-f4x8d","openshift-network-diagnostics/network-check-target-l6shq","kube-system/konnectivity-agent-kmw85","openshift-cluster-node-tuning-operator/tuned-fngdj"]
Jul 28 00:46:37.570584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.570561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.572925 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.572904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.574589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.574252    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.574589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.574304    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 00:46:37.574589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.574357    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.574589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.574307    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5ct4w\""
Jul 28 00:46:37.576893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.575520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.576893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.576346    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 00:46:37.576893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.576614    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-29xrf\""
Jul 28 00:46:37.576893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.576687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 00:46:37.578010 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.577844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.578010 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.577947    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.578517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.578497    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.578937 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.578918    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4dgnx\""
Jul 28 00:46:37.579092 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.578918    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 00:46:37.580699 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.580562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.580699 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.580636    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:37.581458 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.581438    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.581744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.581723    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 00:46:37.581981 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.581960    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 00:46:37.582051 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.582014    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.582210 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.582190    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 00:46:37.582281 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.582236    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 00:46:37.582281 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.582245    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fvglm\""
Jul 28 00:46:37.585976 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.585956    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.586104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.586087    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.586589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.586570    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 00:46:37.588080 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.588061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.589551 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.589530    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.589723 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.589699    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fdn7s\""
Jul 28 00:46:37.589805 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.589788    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.592637 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.591823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.592839 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.592816    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.593259 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.593239    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xt8b5\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.594033    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.594123    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bzjmc\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.594329    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.594787    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.595214    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.595577    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 00:46:37.595888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.595708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.597980 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.597957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysconfig\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.598084 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0481d26-f266-4174-9960-1d2cb5bb6acc-iptables-alerter-script\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.598084 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598052    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-kubelet\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598187 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598086    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-etc-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598187 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysctl-d\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.598318 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-socket-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.598318 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598426 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598320    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fb96e-52a8-4ebd-9333-7850f1805504-konnectivity-ca\") pod \"konnectivity-agent-kmw85\" (UID: \"8a3fb96e-52a8-4ebd-9333-7850f1805504\") " pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.598426 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-etc-selinux\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.598426 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-var-lib-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598574 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-cni-bin\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598574 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598451    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 00:46:37.598574 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598475    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovn-node-metrics-cert\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598574 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a3fb96e-52a8-4ebd-9333-7850f1805504-agent-certs\") pod \"konnectivity-agent-kmw85\" (UID: \"8a3fb96e-52a8-4ebd-9333-7850f1805504\") " pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.598853 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-registration-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.598853 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598639    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-systemd-units\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598853 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovnkube-script-lib\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.598853 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-tuned\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599068 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458525ef-a9ff-4fd0-a301-8b4423fc08f1-tmp\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599068 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598906    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-sys-fs\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.599068 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr5kj\" (UniqueName: \"kubernetes.io/projected/d0481d26-f266-4174-9960-1d2cb5bb6acc-kube-api-access-zr5kj\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.599068 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.598981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.599068 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-modprobe-d\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599068 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr5pw\" (UniqueName: \"kubernetes.io/projected/458525ef-a9ff-4fd0-a301-8b4423fc08f1-kube-api-access-cr5pw\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599093    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5rb6\" (UniqueName: \"kubernetes.io/projected/d3121109-447e-4c0e-a7d5-1a312109d5ba-kube-api-access-f5rb6\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5cd31cd7-5079-487f-9a62-06a8854a3f58-kubelet-config\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599153    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-systemd\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599186    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-ovn\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599202    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vzxbh\""
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599219    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-cni-netd\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-env-overrides\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-kubernetes\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0481d26-f266-4174-9960-1d2cb5bb6acc-host-slash\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599404    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-node-log\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599435    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692b3348-2671-407a-9d86-fa70cdf82395-hosts-file\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599461    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5cd31cd7-5079-487f-9a62-06a8854a3f58-dbus\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3121109-447e-4c0e-a7d5-1a312109d5ba-serviceca\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvb28\" (UniqueName: \"kubernetes.io/projected/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-kube-api-access-nvb28\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-systemd\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-run\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.599910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599865    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-slash\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.599964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frw7t\" (UniqueName: \"kubernetes.io/projected/2bb9b60e-3445-4dfd-b912-70e52b76bf97-kube-api-access-frw7t\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/692b3348-2671-407a-9d86-fa70cdf82395-kube-api-access-2gw6b\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-lib-modules\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-run-netns\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600254    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-run-ovn-kubernetes\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-host\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.600383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3121109-447e-4c0e-a7d5-1a312109d5ba-host\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-log-socket\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-sys\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-var-lib-kubelet\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600581    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovnkube-config\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysctl-conf\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-device-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.600744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.600724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b3348-2671-407a-9d86-fa70cdf82395-tmp-dir\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.602792 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.602171    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.602792 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.602605    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 00:46:37.604588 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.604565    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:37.604708 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.604684    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:37.604981 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.604916    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 00:46:37.605071 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.605027    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nskxd\""
Jul 28 00:46:37.605334 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.605315    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 00:46:37.609177 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.609155    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:37.609306 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.609223    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:37.618691 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.618666    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qx66j"
Jul 28 00:46:37.628182 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.628159    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qx66j"
Jul 28 00:46:37.684990 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.684968    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 00:46:37.701290 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-env-overrides\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.701399 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.701399 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701329    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-kubernetes\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.701399 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0481d26-f266-4174-9960-1d2cb5bb6acc-host-slash\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.701399 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-node-log\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5t2n\" (UniqueName: \"kubernetes.io/projected/2c427abf-95df-4799-a719-3d13f91bd5eb-kube-api-access-t5t2n\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-socket-dir-parent\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-kubernetes\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692b3348-2671-407a-9d86-fa70cdf82395-hosts-file\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d0481d26-f266-4174-9960-1d2cb5bb6acc-host-slash\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-node-log\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.701587    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701607    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/692b3348-2671-407a-9d86-fa70cdf82395-hosts-file\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701619    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5cd31cd7-5079-487f-9a62-06a8854a3f58-dbus\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.701663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701659    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.701674    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:38.201629244 +0000 UTC m=+2.950767950 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3121109-447e-4c0e-a7d5-1a312109d5ba-serviceca\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvb28\" (UniqueName: \"kubernetes.io/projected/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-kube-api-access-nvb28\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701755    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05561ac3-c638-4293-9448-cadf817e389a-multus-daemon-config\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-systemd\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5cd31cd7-5079-487f-9a62-06a8854a3f58-dbus\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-env-overrides\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-systemd\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701965    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-run\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.701996    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-os-release\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-slash\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frw7t\" (UniqueName: \"kubernetes.io/projected/2bb9b60e-3445-4dfd-b912-70e52b76bf97-kube-api-access-frw7t\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/692b3348-2671-407a-9d86-fa70cdf82395-kube-api-access-2gw6b\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-run\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.702208 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-slash\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702107    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-k8s-cni-cncf-io\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-lib-modules\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-run-netns\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702190    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-run-ovn-kubernetes\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-multus-certs\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-host\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702282    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3121109-447e-4c0e-a7d5-1a312109d5ba-host\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-log-socket\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702332    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km68j\" (UniqueName: \"kubernetes.io/projected/3c50f947-74a7-4285-adaa-0e77f24fc849-kube-api-access-km68j\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d3121109-447e-4c0e-a7d5-1a312109d5ba-serviceca\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-lib-modules\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702367    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-system-cni-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-log-socket\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702415    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-run-netns\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-run-ovn-kubernetes\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702424    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-cni-bin\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702454    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d3121109-447e-4c0e-a7d5-1a312109d5ba-host\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.703013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-host\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-sys\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-var-lib-kubelet\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovnkube-config\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-var-lib-kubelet\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702643    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-sys\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-conf-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysctl-conf\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-device-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b3348-2671-407a-9d86-fa70cdf82395-tmp-dir\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-hostroot\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-etc-kubernetes\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysctl-conf\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-device-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysconfig\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0481d26-f266-4174-9960-1d2cb5bb6acc-iptables-alerter-script\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.703959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-kubelet\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.702998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-etc-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-system-cni-dir\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703031    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovnkube-config\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703047    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703071    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-cnibin\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05561ac3-c638-4293-9448-cadf817e389a-cni-binary-copy\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysconfig\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703128    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysctl-d\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-socket-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-kubelet\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703185    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703214    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-etc-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-netns\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703241    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692b3348-2671-407a-9d86-fa70cdf82395-tmp-dir\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-cni-multus\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.704856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703410    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-sysctl-d\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcrpd\" (UniqueName: \"kubernetes.io/projected/05561ac3-c638-4293-9448-cadf817e389a-kube-api-access-gcrpd\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703468    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-socket-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703511    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fb96e-52a8-4ebd-9333-7850f1805504-konnectivity-ca\") pod \"konnectivity-agent-kmw85\" (UID: \"8a3fb96e-52a8-4ebd-9333-7850f1805504\") " pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-etc-selinux\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d0481d26-f266-4174-9960-1d2cb5bb6acc-iptables-alerter-script\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-var-lib-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-cni-bin\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703652    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovn-node-metrics-cert\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a3fb96e-52a8-4ebd-9333-7850f1805504-agent-certs\") pod \"konnectivity-agent-kmw85\" (UID: \"8a3fb96e-52a8-4ebd-9333-7850f1805504\") " pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703686    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-var-lib-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-registration-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-systemd-units\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-cni-bin\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovnkube-script-lib\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.705649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-cnibin\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703861    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-cni-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-kubelet\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-tuned\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.703970    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458525ef-a9ff-4fd0-a301-8b4423fc08f1-tmp\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-sys-fs\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr5kj\" (UniqueName: \"kubernetes.io/projected/d0481d26-f266-4174-9960-1d2cb5bb6acc-kube-api-access-zr5kj\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704057    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704086    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-os-release\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-modprobe-d\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704135    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr5pw\" (UniqueName: \"kubernetes.io/projected/458525ef-a9ff-4fd0-a301-8b4423fc08f1-kube-api-access-cr5pw\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704155    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8a3fb96e-52a8-4ebd-9333-7850f1805504-konnectivity-ca\") pod \"konnectivity-agent-kmw85\" (UID: \"8a3fb96e-52a8-4ebd-9333-7850f1805504\") " pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704175    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5rb6\" (UniqueName: \"kubernetes.io/projected/d3121109-447e-4c0e-a7d5-1a312109d5ba-kube-api-access-f5rb6\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704205    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704232    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5cd31cd7-5079-487f-9a62-06a8854a3f58-kubelet-config\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-systemd\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.706452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-etc-selinux\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-ovn\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-cni-netd\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-registration-dir\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-host-cni-netd\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-openvswitch\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-systemd-units\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-systemd\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2bb9b60e-3445-4dfd-b912-70e52b76bf97-run-ovn\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-modprobe-d\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-sys-fs\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.704922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovnkube-script-lib\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.707293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.705030    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5cd31cd7-5079-487f-9a62-06a8854a3f58-kubelet-config\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:37.709334 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.709042    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2bb9b60e-3445-4dfd-b912-70e52b76bf97-ovn-node-metrics-cert\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.709334 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.709211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8a3fb96e-52a8-4ebd-9333-7850f1805504-agent-certs\") pod \"konnectivity-agent-kmw85\" (UID: \"8a3fb96e-52a8-4ebd-9333-7850f1805504\") " pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.709334 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.709271    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/458525ef-a9ff-4fd0-a301-8b4423fc08f1-etc-tuned\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.710131 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.710105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/458525ef-a9ff-4fd0-a301-8b4423fc08f1-tmp\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.715674 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.715647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frw7t\" (UniqueName: \"kubernetes.io/projected/2bb9b60e-3445-4dfd-b912-70e52b76bf97-kube-api-access-frw7t\") pod \"ovnkube-node-j849f\" (UID: \"2bb9b60e-3445-4dfd-b912-70e52b76bf97\") " pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.716176 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.716150    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvb28\" (UniqueName: \"kubernetes.io/projected/9c149f26-4ef4-42bc-bf7d-4c9aad5b293e-kube-api-access-nvb28\") pod \"aws-ebs-csi-driver-node-n9vqq\" (UID: \"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.717143 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.717104    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5rb6\" (UniqueName: \"kubernetes.io/projected/d3121109-447e-4c0e-a7d5-1a312109d5ba-kube-api-access-f5rb6\") pod \"node-ca-2575g\" (UID: \"d3121109-447e-4c0e-a7d5-1a312109d5ba\") " pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.717608 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.717529    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr5kj\" (UniqueName: \"kubernetes.io/projected/d0481d26-f266-4174-9960-1d2cb5bb6acc-kube-api-access-zr5kj\") pod \"iptables-alerter-rsltp\" (UID: \"d0481d26-f266-4174-9960-1d2cb5bb6acc\") " pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.718675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.718626    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr5pw\" (UniqueName: \"kubernetes.io/projected/458525ef-a9ff-4fd0-a301-8b4423fc08f1-kube-api-access-cr5pw\") pod \"tuned-fngdj\" (UID: \"458525ef-a9ff-4fd0-a301-8b4423fc08f1\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.719184 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.719155    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gw6b\" (UniqueName: \"kubernetes.io/projected/692b3348-2671-407a-9d86-fa70cdf82395-kube-api-access-2gw6b\") pod \"node-resolver-z427m\" (UID: \"692b3348-2671-407a-9d86-fa70cdf82395\") " pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.749568 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.749513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" event={"ID":"49d5be5d46952d796bae73c6a91a14e0","Type":"ContainerStarted","Data":"44346a4ae7855c9bf5c3a37a721d58093b4330d6aeba6751e67b44fead687649"}
Jul 28 00:46:37.750752 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.750705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal" event={"ID":"851ad20362c20c22501be689ae7a68aa","Type":"ContainerStarted","Data":"716a699d2e0461af373faf4684934bb0a4c4256307f2886eb4e7f62d0c5b8724"}
Jul 28 00:46:37.805304 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805272    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-hostroot\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805428 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-etc-kubernetes\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805428 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-system-cni-dir\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.805428 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-system-cni-dir\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.805428 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805403    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-etc-kubernetes\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805671 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805412    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-hostroot\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805671 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805438    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.805671 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-cnibin\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805671 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805530    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05561ac3-c638-4293-9448-cadf817e389a-cni-binary-copy\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805671 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:37.805671 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-cnibin\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-netns\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805690    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-netns\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.805728    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-cni-multus\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcrpd\" (UniqueName: \"kubernetes.io/projected/05561ac3-c638-4293-9448-cadf817e389a-kube-api-access-gcrpd\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.805795    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:38.305773546 +0000 UTC m=+3.054912263 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805869    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-cnibin\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.805918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-cni-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-kubelet\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-os-release\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.805980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-cni-multus\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5t2n\" (UniqueName: \"kubernetes.io/projected/2c427abf-95df-4799-a719-3d13f91bd5eb-kube-api-access-t5t2n\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-socket-dir-parent\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05561ac3-c638-4293-9448-cadf817e389a-multus-daemon-config\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-os-release\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-k8s-cni-cncf-io\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/05561ac3-c638-4293-9448-cadf817e389a-cni-binary-copy\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806229    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-multus-certs\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-socket-dir-parent\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-multus-certs\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-kubelet\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806387    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km68j\" (UniqueName: \"kubernetes.io/projected/3c50f947-74a7-4285-adaa-0e77f24fc849-kube-api-access-km68j\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-run-k8s-cni-cncf-io\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806416    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-system-cni-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806455    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-system-cni-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-cni-bin\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-cnibin\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806499    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-conf-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806544    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-cni-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806586    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2c427abf-95df-4799-a719-3d13f91bd5eb-os-release\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-os-release\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.806793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-host-var-lib-cni-bin\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.807584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/05561ac3-c638-4293-9448-cadf817e389a-multus-conf-dir\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.807584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806933    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-cni-binary-copy\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.807584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.806992    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.807584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.807101    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/05561ac3-c638-4293-9448-cadf817e389a-multus-daemon-config\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.807584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.807143    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2c427abf-95df-4799-a719-3d13f91bd5eb-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.814139 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.814083    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:37.814139 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.814110    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:37.814139 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.814124    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:37.814339 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:37.814203    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:38.314185316 +0000 UTC m=+3.063324021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:37.816280 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.816255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km68j\" (UniqueName: \"kubernetes.io/projected/3c50f947-74a7-4285-adaa-0e77f24fc849-kube-api-access-km68j\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:37.817252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.817226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcrpd\" (UniqueName: \"kubernetes.io/projected/05561ac3-c638-4293-9448-cadf817e389a-kube-api-access-gcrpd\") pod \"multus-2fvjp\" (UID: \"05561ac3-c638-4293-9448-cadf817e389a\") " pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.817252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.817244    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5t2n\" (UniqueName: \"kubernetes.io/projected/2c427abf-95df-4799-a719-3d13f91bd5eb-kube-api-access-t5t2n\") pod \"multus-additional-cni-plugins-frfxj\" (UID: \"2c427abf-95df-4799-a719-3d13f91bd5eb\") " pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:37.859552 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.859522    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:37.888276 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.888247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq"
Jul 28 00:46:37.902493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.902469    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:46:37.914606 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.914579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rsltp"
Jul 28 00:46:37.921476 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.921441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:46:37.929328 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.929310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fngdj"
Jul 28 00:46:37.937156 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.937138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z427m"
Jul 28 00:46:37.945870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.945853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2575g"
Jul 28 00:46:37.960697 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.960674    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2fvjp"
Jul 28 00:46:37.968465 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:37.968448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frfxj"
Jul 28 00:46:38.210981 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:38.210890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:38.211140 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.211035    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:38.211140 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.211103    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:39.211088553 +0000 UTC m=+3.960227278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:38.312239 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:38.312194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:38.312405 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.312333    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:38.312405 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.312404    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:39.312386869 +0000 UTC m=+4.061525574 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:38.413302 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:38.413265    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:38.413481 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.413387    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:38.413481 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.413407    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:38.413481 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.413418    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:38.413481 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:38.413474    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:39.413455379 +0000 UTC m=+4.162594101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:38.631519 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:38.631481    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:41:38 +0000 UTC" deadline="2028-03-18 14:28:45.661192815 +0000 UTC"
Jul 28 00:46:38.631519 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:38.631513    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14389h42m7.029684046s"
Jul 28 00:46:38.814647 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.814616    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c427abf_95df_4799_a719_3d13f91bd5eb.slice/crio-de83abac39493c6f539e5f22e5e66172db397ff5a41616fe53258bd809ce39d8 WatchSource:0}: Error finding container de83abac39493c6f539e5f22e5e66172db397ff5a41616fe53258bd809ce39d8: Status 404 returned error can't find the container with id de83abac39493c6f539e5f22e5e66172db397ff5a41616fe53258bd809ce39d8
Jul 28 00:46:38.819715 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.819687    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3121109_447e_4c0e_a7d5_1a312109d5ba.slice/crio-625a786e8c79e48ffaa39b76abdfabde1c381faec414018766a1713cbd4cb5ab WatchSource:0}: Error finding container 625a786e8c79e48ffaa39b76abdfabde1c381faec414018766a1713cbd4cb5ab: Status 404 returned error can't find the container with id 625a786e8c79e48ffaa39b76abdfabde1c381faec414018766a1713cbd4cb5ab
Jul 28 00:46:38.821647 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.821611    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c149f26_4ef4_42bc_bf7d_4c9aad5b293e.slice/crio-ebc558d7a2f83b0bb7b2efb7ddff315be6dca91257088ff66f05735f1e00b04f WatchSource:0}: Error finding container ebc558d7a2f83b0bb7b2efb7ddff315be6dca91257088ff66f05735f1e00b04f: Status 404 returned error can't find the container with id ebc558d7a2f83b0bb7b2efb7ddff315be6dca91257088ff66f05735f1e00b04f
Jul 28 00:46:38.823196 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.822861    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb9b60e_3445_4dfd_b912_70e52b76bf97.slice/crio-7ae8c8417d47fbf0fd99a613de50b458af7788fc2253a8030440ec591fa0eefe WatchSource:0}: Error finding container 7ae8c8417d47fbf0fd99a613de50b458af7788fc2253a8030440ec591fa0eefe: Status 404 returned error can't find the container with id 7ae8c8417d47fbf0fd99a613de50b458af7788fc2253a8030440ec591fa0eefe
Jul 28 00:46:38.823819 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.823790    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692b3348_2671_407a_9d86_fa70cdf82395.slice/crio-43ef4572000612a27e0a6081f2bc399f3e2722a62a28c3bc0e6ecfa4a2fed4fc WatchSource:0}: Error finding container 43ef4572000612a27e0a6081f2bc399f3e2722a62a28c3bc0e6ecfa4a2fed4fc: Status 404 returned error can't find the container with id 43ef4572000612a27e0a6081f2bc399f3e2722a62a28c3bc0e6ecfa4a2fed4fc
Jul 28 00:46:38.824460 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.824395    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3fb96e_52a8_4ebd_9333_7850f1805504.slice/crio-59fdc89447ee2f43e203c39c7eed7aa4a2c81814c3f8790c9dedab8554e4c850 WatchSource:0}: Error finding container 59fdc89447ee2f43e203c39c7eed7aa4a2c81814c3f8790c9dedab8554e4c850: Status 404 returned error can't find the container with id 59fdc89447ee2f43e203c39c7eed7aa4a2c81814c3f8790c9dedab8554e4c850
Jul 28 00:46:38.826446 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.826317    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05561ac3_c638_4293_9448_cadf817e389a.slice/crio-4826df1f97af035787a1563378ceb3930c02d5204d81948648232e15e0daa05c WatchSource:0}: Error finding container 4826df1f97af035787a1563378ceb3930c02d5204d81948648232e15e0daa05c: Status 404 returned error can't find the container with id 4826df1f97af035787a1563378ceb3930c02d5204d81948648232e15e0daa05c
Jul 28 00:46:38.827652 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.826974    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0481d26_f266_4174_9960_1d2cb5bb6acc.slice/crio-bb26320527ddc5f20e2f1f2c0fa789fd4dc54cdf53a810d98b41efccba3d8cd0 WatchSource:0}: Error finding container bb26320527ddc5f20e2f1f2c0fa789fd4dc54cdf53a810d98b41efccba3d8cd0: Status 404 returned error can't find the container with id bb26320527ddc5f20e2f1f2c0fa789fd4dc54cdf53a810d98b41efccba3d8cd0
Jul 28 00:46:38.827928 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:46:38.827812    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458525ef_a9ff_4fd0_a301_8b4423fc08f1.slice/crio-ddb775f54c8660b92da4a1b41750933a5859e86846a824154e997749905fb982 WatchSource:0}: Error finding container ddb775f54c8660b92da4a1b41750933a5859e86846a824154e997749905fb982: Status 404 returned error can't find the container with id ddb775f54c8660b92da4a1b41750933a5859e86846a824154e997749905fb982
Jul 28 00:46:38.911030 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:38.911006    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 00:46:39.219104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.218847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:39.219247 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.219005    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:39.219247 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.219215    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:41.219198319 +0000 UTC m=+5.968337025 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:39.320358 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.320317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:39.320545 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.320527    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:39.320623 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.320593    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:41.320573303 +0000 UTC m=+6.069712019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:39.421445 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.421404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:39.421630 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.421610    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:39.421630 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.421628    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:39.421750 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.421642    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:39.421750 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.421704    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:41.421685608 +0000 UTC m=+6.170824313 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:39.631960 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.631858    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 00:41:38 +0000 UTC" deadline="2028-04-03 21:53:42.40809448 +0000 UTC"
Jul 28 00:46:39.631960 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.631958    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14781h7m2.77614118s"
Jul 28 00:46:39.744618 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.744584    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:39.744808 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.744725    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:39.745462 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.745170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:39.745462 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.745274    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:39.745462 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.745334    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:39.745462 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:39.745425    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:39.761556 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.761526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2fvjp" event={"ID":"05561ac3-c638-4293-9448-cadf817e389a","Type":"ContainerStarted","Data":"4826df1f97af035787a1563378ceb3930c02d5204d81948648232e15e0daa05c"}
Jul 28 00:46:39.766622 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.766594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kmw85" event={"ID":"8a3fb96e-52a8-4ebd-9333-7850f1805504","Type":"ContainerStarted","Data":"59fdc89447ee2f43e203c39c7eed7aa4a2c81814c3f8790c9dedab8554e4c850"}
Jul 28 00:46:39.775478 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.775341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"7ae8c8417d47fbf0fd99a613de50b458af7788fc2253a8030440ec591fa0eefe"}
Jul 28 00:46:39.794440 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.794349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq" event={"ID":"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e","Type":"ContainerStarted","Data":"ebc558d7a2f83b0bb7b2efb7ddff315be6dca91257088ff66f05735f1e00b04f"}
Jul 28 00:46:39.801312 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.801115    2576 generic.go:356] "Generic (PLEG): container finished" podID="49d5be5d46952d796bae73c6a91a14e0" containerID="3f69dd97f5f5d0aac3c11db10e440633e74ecc0bd0bb2b5a8d4a1e4cbc66ec36" exitCode=0
Jul 28 00:46:39.801312 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.801159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" event={"ID":"49d5be5d46952d796bae73c6a91a14e0","Type":"ContainerDied","Data":"3f69dd97f5f5d0aac3c11db10e440633e74ecc0bd0bb2b5a8d4a1e4cbc66ec36"}
Jul 28 00:46:39.813885 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.813795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsltp" event={"ID":"d0481d26-f266-4174-9960-1d2cb5bb6acc","Type":"ContainerStarted","Data":"bb26320527ddc5f20e2f1f2c0fa789fd4dc54cdf53a810d98b41efccba3d8cd0"}
Jul 28 00:46:39.819376 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.819327    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z427m" event={"ID":"692b3348-2671-407a-9d86-fa70cdf82395","Type":"ContainerStarted","Data":"43ef4572000612a27e0a6081f2bc399f3e2722a62a28c3bc0e6ecfa4a2fed4fc"}
Jul 28 00:46:39.838020 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.837942    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2575g" event={"ID":"d3121109-447e-4c0e-a7d5-1a312109d5ba","Type":"ContainerStarted","Data":"625a786e8c79e48ffaa39b76abdfabde1c381faec414018766a1713cbd4cb5ab"}
Jul 28 00:46:39.850709 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.850682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerStarted","Data":"de83abac39493c6f539e5f22e5e66172db397ff5a41616fe53258bd809ce39d8"}
Jul 28 00:46:39.859822 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.859228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal" event={"ID":"851ad20362c20c22501be689ae7a68aa","Type":"ContainerStarted","Data":"760efae426f303c8b68284a0a16b1f8238a1be13e5b32382b4651eb2ce0f0127"}
Jul 28 00:46:39.875730 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.875670    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-154.ec2.internal" podStartSLOduration=2.875652338 podStartE2EDuration="2.875652338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:39.875132643 +0000 UTC m=+4.624271370" watchObservedRunningTime="2026-07-28 00:46:39.875652338 +0000 UTC m=+4.624791067"
Jul 28 00:46:39.876197 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:39.876139    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fngdj" event={"ID":"458525ef-a9ff-4fd0-a301-8b4423fc08f1","Type":"ContainerStarted","Data":"ddb775f54c8660b92da4a1b41750933a5859e86846a824154e997749905fb982"}
Jul 28 00:46:40.894972 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:40.894934    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" event={"ID":"49d5be5d46952d796bae73c6a91a14e0","Type":"ContainerStarted","Data":"d9397a1ea9fa386c4cc6d1212729348a0ecdb3942f4ba773148a77abc2436c83"}
Jul 28 00:46:40.915049 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:40.914593    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-154.ec2.internal" podStartSLOduration=3.914568374 podStartE2EDuration="3.914568374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:40.913674631 +0000 UTC m=+5.662813360" watchObservedRunningTime="2026-07-28 00:46:40.914568374 +0000 UTC m=+5.663707102"
Jul 28 00:46:41.239582 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:41.239486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:41.239754 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.239647    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:41.239754 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.239718    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.239697431 +0000 UTC m=+9.988836150 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:41.340225 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:41.340184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:41.340426 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.340406    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:41.340499 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.340481    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.340451891 +0000 UTC m=+10.089590600 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:41.441440 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:41.441397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:41.454403 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.454368    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:41.454403 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.454398    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:41.454627 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.454435    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:41.454627 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.454513    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:45.454482868 +0000 UTC m=+10.203621597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:41.747048 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:41.747012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:41.747247 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.747155    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:41.747609 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:41.747588    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:41.747708 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.747693    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:41.747807 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:41.747765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:41.747873 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:41.747856    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:43.750495 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:43.750472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:43.750869 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:43.750589    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:43.750869 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:43.750631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:43.750869 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:43.750674    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:43.750869 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:43.750700    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:43.750869 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:43.750744    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:45.277073 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:45.277030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:45.277596 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.277221    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:45.277596 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.277324    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:53.277299564 +0000 UTC m=+18.026438274 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:45.378207 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:45.378167    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:45.378418 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.378351    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:45.378418 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.378416    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:53.378400596 +0000 UTC m=+18.127539300 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:45.479170 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:45.479125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:45.479354 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.479330    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:45.479354 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.479351    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:45.479482 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.479361    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:45.479482 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.479419    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:46:53.479400207 +0000 UTC m=+18.228538915 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:45.749617 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:45.749573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:45.749812 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.749741    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:45.750252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:45.750226    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:45.750404 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.750383    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:45.751386 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:45.751369    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:45.751542 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:45.751523    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:47.745413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:47.745383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:47.745888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:47.745395    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:47.745888 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:47.745506    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:47.745888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:47.745520    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:47.745888 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:47.745600    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:47.745888 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:47.745699    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:49.745356 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:49.745329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:49.745356 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:49.745329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:49.745844 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:49.745336    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:49.745844 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:49.745438    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:49.745844 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:49.745524    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:49.745844 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:49.745613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:51.744869 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:51.744833    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:51.744869 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:51.744842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:51.745413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:51.744951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:51.745413 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:51.744961    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:51.745413 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:51.745064    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:51.745413 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:51.745150    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:53.340232 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:53.340186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:53.340654 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.340338    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:53.340654 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.340408    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:09.340389884 +0000 UTC m=+34.089528586 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:46:53.440630 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:53.440598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:53.440801 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.440779    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:53.440861 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.440855    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:09.440834148 +0000 UTC m=+34.189972873 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:46:53.541696 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:53.541661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:53.541855 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.541804    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:46:53.541855 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.541824    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:46:53.541855 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.541835    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:53.542004 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.541893    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:09.541875583 +0000 UTC m=+34.291014300 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:46:53.744570 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:53.744501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:53.744570 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:53.744528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:53.744756 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.744593    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:53.744756 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.744648    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:53.744756 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:53.744660    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:53.744756 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:53.744733    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:55.747238 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:55.747201    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:55.747652 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:55.747323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:55.747652 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:55.747327    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:55.747652 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:55.747368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:55.747652 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:55.747469    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:55.747652 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:55.747544    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:57.749392 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.749370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:57.749761 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:57.749479    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:57.749830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.749778    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:57.749884 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:57.749866    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:57.749934 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.749917    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:57.750014 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:57.749994    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:57.948931 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.948740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2575g" event={"ID":"d3121109-447e-4c0e-a7d5-1a312109d5ba","Type":"ContainerStarted","Data":"40de0d02d02291e3bcf0401491d935942fcd4ef6c6fab588e1a5a42ed9390720"}
Jul 28 00:46:57.950655 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.950609    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerStarted","Data":"1049bcafa798f6fba48266ca529a393c1b8cc665c9f42367e2988edf3e636a54"}
Jul 28 00:46:57.952524 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.952473    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fngdj" event={"ID":"458525ef-a9ff-4fd0-a301-8b4423fc08f1","Type":"ContainerStarted","Data":"86b1b3da144b0a7a5d1a22080ff140a6a4aba47d949a1d3cc2be856249719aba"}
Jul 28 00:46:57.953785 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.953760    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2fvjp" event={"ID":"05561ac3-c638-4293-9448-cadf817e389a","Type":"ContainerStarted","Data":"45efcb7b4329a5b6aefb8c3936e38b8fc5be9fbe03324145347c36338d6f4f91"}
Jul 28 00:46:57.955655 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.955634    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kmw85" event={"ID":"8a3fb96e-52a8-4ebd-9333-7850f1805504","Type":"ContainerStarted","Data":"0d43fdbc72e1f75398569a0454124c53e7f18eb36cc6855cf076f95377c10bd4"}
Jul 28 00:46:57.957458 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.957438    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"87ea102b34a618650a9ad8f941a1dc6ddc24b88554a208c37127755e454a959b"}
Jul 28 00:46:57.959252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.959208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq" event={"ID":"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e","Type":"ContainerStarted","Data":"0ccc21007b76d19b6ded6dbd8eb259690d4472fe7e4603003fda126e060ba520"}
Jul 28 00:46:57.960604 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.960333    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z427m" event={"ID":"692b3348-2671-407a-9d86-fa70cdf82395","Type":"ContainerStarted","Data":"5ad1237066aec87466f41288797a4398440e3ea223985ca4d8e2058e8baf5b74"}
Jul 28 00:46:57.963073 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.962788    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2575g" podStartSLOduration=3.07165927 podStartE2EDuration="21.962776913s" totalImagesPullingTime="18.891117643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:57.962332802 +0000 UTC m=+22.711471526" watchObservedRunningTime="2026-07-28 00:46:57.962776913 +0000 UTC m=+22.711915636"
Jul 28 00:46:57.978786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.978747    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kmw85" podStartSLOduration=10.911221265 podStartE2EDuration="21.978735085s" totalImagesPullingTime="11.06751382s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:57.978380501 +0000 UTC m=+22.727519224" watchObservedRunningTime="2026-07-28 00:46:57.978735085 +0000 UTC m=+22.727873835"
Jul 28 00:46:57.993734 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:57.993690    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-z427m" podStartSLOduration=3.107168349 podStartE2EDuration="21.99367847s" totalImagesPullingTime="18.886510121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:57.993093421 +0000 UTC m=+22.742232145" watchObservedRunningTime="2026-07-28 00:46:57.99367847 +0000 UTC m=+22.742817193"
Jul 28 00:46:58.046285 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.046243    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fngdj" podStartSLOduration=3.187303112 podStartE2EDuration="22.046205465s" totalImagesPullingTime="18.858902353s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:58.030116018 +0000 UTC m=+22.779254744" watchObservedRunningTime="2026-07-28 00:46:58.046205465 +0000 UTC m=+22.795344189"
Jul 28 00:46:58.046417 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.046372    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2fvjp" podStartSLOduration=3.13482683 podStartE2EDuration="22.046363052s" totalImagesPullingTime="18.911536222s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:58.046035968 +0000 UTC m=+22.795174898" watchObservedRunningTime="2026-07-28 00:46:58.046363052 +0000 UTC m=+22.795501777"
Jul 28 00:46:58.406128 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.406095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:58.588892 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.588857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:58.742681 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.742652    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:58.843196 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.843168    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:58.964753 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.964725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"bbd3d90e49ca1b713abbf160e665c9c042e5604d47a7f4dd3c015ce45f58a729"}
Jul 28 00:46:58.964878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.964833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"7022aef0f35f14dd3d1e3ea80d42263fab14cb4edb10411f3ac3d8f22fa2f38a"}
Jul 28 00:46:58.964878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.964870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"5d5a5354538a2f7c6cb5ab1f77cc9da73245c61969b15df1d5fe66faa3d305e7"}
Jul 28 00:46:58.964978 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.964904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"b90da5b88cf7e8e22f6ce3dc9d97cff1c5ff6d3c72209d5f92b6f401971814bc"}
Jul 28 00:46:58.964978 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.964948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"d23de8de3eafb526401e05895a62b443074cc9f1362bb635544851bdb3085945"}
Jul 28 00:46:58.965970 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.965946    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c427abf-95df-4799-a719-3d13f91bd5eb" containerID="1049bcafa798f6fba48266ca529a393c1b8cc665c9f42367e2988edf3e636a54" exitCode=0
Jul 28 00:46:58.966070 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:58.966039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerDied","Data":"1049bcafa798f6fba48266ca529a393c1b8cc665c9f42367e2988edf3e636a54"}
Jul 28 00:46:59.040290 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.040269    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.141418 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.141390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.166373 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.166351    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 00:46:59.241909 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.241883    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.664091 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.663943    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T00:46:59.166369569Z","UUID":"80755c14-e166-4a7f-ab75-928dbe8fae42","Handler":null,"Name":"","Endpoint":""}
Jul 28 00:46:59.665668 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.665622    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 00:46:59.665668 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.665666    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 00:46:59.709659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.709629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.748531 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.748507    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:46:59.748531 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.748528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:46:59.748715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.748507    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:46:59.748715 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:59.748613    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:46:59.748715 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:59.748706    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:46:59.748864 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:46:59.748767    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:46:59.809128 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.809043    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.909313 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.909258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.965451 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.965390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:46:59.972057 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.972029    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rsltp" event={"ID":"d0481d26-f266-4174-9960-1d2cb5bb6acc","Type":"ContainerStarted","Data":"a930fa13acc81bd9c58d4fc56daa03bd924170bb763116e49798aa9c8eff1e7e"}
Jul 28 00:46:59.974291 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.974259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq" event={"ID":"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e","Type":"ContainerStarted","Data":"bbceb17293016a624af0b0759d80df5eb1c74169bb90f7d0c26c809e4be916b8"}
Jul 28 00:46:59.988699 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:46:59.988656    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rsltp" podStartSLOduration=5.133978529 podStartE2EDuration="23.988642836s" totalImagesPullingTime="18.854664307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:46:59.988612511 +0000 UTC m=+24.737751236" watchObservedRunningTime="2026-07-28 00:46:59.988642836 +0000 UTC m=+24.737781559"
Jul 28 00:47:00.014706 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:00.014685    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:47:00.898730 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:00.898698    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:00.981546 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:00.981483    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"c06abb8752de67e4bd337bbbf9f0a5c6931a3d8902f177761b8364351542052d"}
Jul 28 00:47:00.984024 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:00.983958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq" event={"ID":"9c149f26-4ef4-42bc-bf7d-4c9aad5b293e","Type":"ContainerStarted","Data":"8e6f13a87b7f7db8f82743aba4d0d73ae050aaf79fc33219d7e43603c28a6d95"}
Jul 28 00:47:01.002756 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.002710    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-n9vqq" podStartSLOduration=3.678332062 podStartE2EDuration="25.002691606s" totalImagesPullingTime="21.324359544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:01.002270057 +0000 UTC m=+25.751408783" watchObservedRunningTime="2026-07-28 00:47:01.002691606 +0000 UTC m=+25.751830327"
Jul 28 00:47:01.111409 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.111381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:01.220460 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.220378    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:47:01.221050 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.221028    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:47:01.299536 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.299503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:01.508442 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.508363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:01.697693 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.697663    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:01.747628 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.747602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:01.747794 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.747708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:01.747794 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:01.747720    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:01.747794 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.747738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:01.747929 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:01.747820    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:01.747929 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:01.747889    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:01.897005 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.896965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:01.986777 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:01.986753    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kmw85"
Jul 28 00:47:02.144930 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:02.144890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:02.418254 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:02.418208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:02.903534 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:02.903495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:03.273001 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.272980    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:03.373148 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.373117    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:03.522830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.522805    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:03.701400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.701122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:03.747153 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.747134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:03.747256 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.747238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:03.747256 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:03.747246    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:03.747342 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.747262    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:03.747376 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:03.747324    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:03.747423 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:03.747407    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:03.900542 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.900520    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:03.991650 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.991591    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c427abf-95df-4799-a719-3d13f91bd5eb" containerID="2c5f86bd9815dbdea7c53397b7a08ab4a62bcec4bddcbde831658d2c5a68b0c7" exitCode=0
Jul 28 00:47:03.991747 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.991676    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerDied","Data":"2c5f86bd9815dbdea7c53397b7a08ab4a62bcec4bddcbde831658d2c5a68b0c7"}
Jul 28 00:47:03.995321 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.995299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" event={"ID":"2bb9b60e-3445-4dfd-b912-70e52b76bf97","Type":"ContainerStarted","Data":"f5229e9ab56981d5c40f41ab0afec5c1fd3ae8836e9d9cccb7acf595bfeccdc8"}
Jul 28 00:47:03.995587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.995565    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:47:03.995680 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:03.995641    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:47:04.010065 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:04.010043    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:47:04.043776 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:04.043738    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-j849f" podStartSLOduration=9.096087089 podStartE2EDuration="28.043725656s" totalImagesPullingTime="18.947638567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:04.042452196 +0000 UTC m=+28.791590920" watchObservedRunningTime="2026-07-28 00:47:04.043725656 +0000 UTC m=+28.792864381"
Jul 28 00:47:04.896191 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:04.896158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:04.997792 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:04.997771    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:47:05.012196 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.012173    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:47:05.303521 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.301441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:05.501413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.501338    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:05.688349 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.688321    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:05.746057 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.746029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:05.746206 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:05.746106    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:05.746206 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.746195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:05.746316 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:05.746298    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:05.746374 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.746362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:05.746434 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:05.746420    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:05.750045 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:05.750022    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:06.001124 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:06.001092    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c427abf-95df-4799-a719-3d13f91bd5eb" containerID="915b7ab345785c54d45e9b79cb6bfb22ec91a549b803d9cc1d75149e6c46da48" exitCode=0
Jul 28 00:47:06.001538 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:06.001165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerDied","Data":"915b7ab345785c54d45e9b79cb6bfb22ec91a549b803d9cc1d75149e6c46da48"}
Jul 28 00:47:06.104158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:06.104137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:06.703336 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:06.703307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:07.005459 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.005376    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c427abf-95df-4799-a719-3d13f91bd5eb" containerID="2336493926d688316c2a40c057ca39599801c27e479656a7b1cb9dc47af6c733" exitCode=0
Jul 28 00:47:07.005459 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.005415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerDied","Data":"2336493926d688316c2a40c057ca39599801c27e479656a7b1cb9dc47af6c733"}
Jul 28 00:47:07.108878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.108834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:07.316013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.315982    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:07.745457 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.745375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:07.745457 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.745401    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:07.745457 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.745376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:07.745744 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:07.745496    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:07.745744 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:07.745653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:07.745744 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:07.745689    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:07.900374 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:07.900337    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:08.363102 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:08.363071    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:08.701981 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:08.701898    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:08.907643 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:08.907604    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:09.101606 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.101565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:09.292995 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.292963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:09.351700 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.351615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:09.360542 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.360514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:09.360658 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.360625    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 00:47:09.360727 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.360684    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret podName:5cd31cd7-5079-487f-9a62-06a8854a3f58 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:41.360666366 +0000 UTC m=+66.109805088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret") pod "global-pull-secret-syncer-zw59q" (UID: "5cd31cd7-5079-487f-9a62-06a8854a3f58") : object "kube-system"/"original-pull-secret" not registered
Jul 28 00:47:09.461354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.461321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:09.461772 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.461458    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:47:09.461772 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.461531    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs podName:3c50f947-74a7-4285-adaa-0e77f24fc849 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:41.461509004 +0000 UTC m=+66.210647728 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs") pod "network-metrics-daemon-f4x8d" (UID: "3c50f947-74a7-4285-adaa-0e77f24fc849") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 00:47:09.562240 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.562192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:09.562425 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.562363    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 00:47:09.562425 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.562390    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 00:47:09.562425 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.562402    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ncfkt for pod openshift-network-diagnostics/network-check-target-l6shq: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:47:09.562556 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.562463    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt podName:8e2d0561-a486-4f89-a50e-9f9e5cbfe537 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:41.562446569 +0000 UTC m=+66.311585292 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ncfkt" (UniqueName: "kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt") pod "network-check-target-l6shq" (UID: "8e2d0561-a486-4f89-a50e-9f9e5cbfe537") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 00:47:09.745205 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.745131    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:09.745392 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.745134    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:09.745392 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.745265    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:09.745392 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:09.745138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:09.745392 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.745363    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:09.745589 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:09.745424    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:11.745377 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:11.745341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:11.745812 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:11.745341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:11.745812 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:11.745449    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:11.745812 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:11.745546    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:11.745812 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:11.745341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:11.745812 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:11.745644    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:12.019569 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:12.019489    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c427abf-95df-4799-a719-3d13f91bd5eb" containerID="fb19bf7bfd59c15f02eccc516ac3a68d8dacbf2002ced46a766c10499dc48a1a" exitCode=0
Jul 28 00:47:12.019569 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:12.019543    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerDied","Data":"fb19bf7bfd59c15f02eccc516ac3a68d8dacbf2002ced46a766c10499dc48a1a"}
Jul 28 00:47:13.024316 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:13.024281    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c427abf-95df-4799-a719-3d13f91bd5eb" containerID="0f8c2efd3a5755c567ee0c9cdb1b7f0af8c590f7f9b337c770aba670ea2e605d" exitCode=0
Jul 28 00:47:13.024670 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:13.024339    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerDied","Data":"0f8c2efd3a5755c567ee0c9cdb1b7f0af8c590f7f9b337c770aba670ea2e605d"}
Jul 28 00:47:13.745083 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:13.745053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:13.745083 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:13.745066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:13.745083 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:13.745078    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:13.745401 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:13.745164    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:13.745401 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:13.745268    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:13.745401 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:13.745356    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:14.028968 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:14.028893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frfxj" event={"ID":"2c427abf-95df-4799-a719-3d13f91bd5eb","Type":"ContainerStarted","Data":"16b65bd44e8b06c61dfd1c8af1c86ffa3402187abee49c0b81f0dd7223677155"}
Jul 28 00:47:14.049935 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:14.049898    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-frfxj" podStartSLOduration=9.017342717 podStartE2EDuration="38.049884388s" totalImagesPullingTime="29.032541671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:14.048375225 +0000 UTC m=+38.797513949" watchObservedRunningTime="2026-07-28 00:47:14.049884388 +0000 UTC m=+38.799023112"
Jul 28 00:47:15.746059 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:15.746026    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:15.746517 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:15.746111    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:15.746517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:15.746156    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:15.746517 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:15.746204    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:15.746517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:15.746268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:15.746517 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:15.746335    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:15.919795 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:15.919762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.024700 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.024632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.123129 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.123103    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.222757 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.222732    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.395485 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.395459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.605203 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.605167    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.705352 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.705292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.915772 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.915738    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:16.999786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:16.999725    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:17.096359 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.096323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:17.174685 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.174657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:17.286596 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.286533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:17.648011 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.647975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:17.745028 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.745002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:17.745178 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.745011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:17.745178 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:17.745146    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:17.745299 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.745011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:17.745299 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:17.745201    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:17.745299 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:17.745291    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:17.972530 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:17.972339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:18.072509 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.072472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:18.489721 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.489691    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zw59q"]
Jul 28 00:47:18.489874 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.489795    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:18.489932 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:18.489903    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:18.492908 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.492885    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f4x8d"]
Jul 28 00:47:18.493150 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.493133    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:18.493410 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:18.493365    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:18.493733 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.493712    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6shq"]
Jul 28 00:47:18.493808 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.493797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:18.493907 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:18.493887    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:18.820285 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.820246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:18.922746 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:18.922718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:19.744924 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:19.744856    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:19.745414 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:19.744957    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:20.745084 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:20.745057    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:20.745489 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:20.745061    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:20.745489 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:20.745147    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zw59q" podUID="5cd31cd7-5079-487f-9a62-06a8854a3f58"
Jul 28 00:47:20.745489 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:20.745233    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-l6shq" podUID="8e2d0561-a486-4f89-a50e-9f9e5cbfe537"
Jul 28 00:47:21.745195 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:21.745165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:21.745616 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:21.745277    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-f4x8d" podUID="3c50f947-74a7-4285-adaa-0e77f24fc849"
Jul 28 00:47:22.271312 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.271285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:22.370215 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.370189    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:22.630909 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.630887    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-154.ec2.internal" event="NodeReady"
Jul 28 00:47:22.631060 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.630984    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 00:47:22.676314 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.676291    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-2nsh7"]
Jul 28 00:47:22.705662 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.705639    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mlmxc"]
Jul 28 00:47:22.705789 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.705744    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.709985 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.709959    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 00:47:22.710102 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.709992    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-54rrh\""
Jul 28 00:47:22.710169 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.710110    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 00:47:22.726841 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.726820    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nsh7"]
Jul 28 00:47:22.726944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.726851    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-bzfc2"]
Jul 28 00:47:22.726944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.726931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.731801 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.731781    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 00:47:22.731923 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.731827    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-nzvbc\""
Jul 28 00:47:22.732675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.732333    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 00:47:22.732675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.732573    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 00:47:22.745408 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.745390    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mlmxc"]
Jul 28 00:47:22.745816 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.745432    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzfc2"]
Jul 28 00:47:22.745816 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.745488    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.745816 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.745501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:22.745816 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.745579    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:22.748945 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.748928    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 00:47:22.749059 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.748942    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 00:47:22.749059 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.748986    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 00:47:22.749893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.749877    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 00:47:22.749893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.749884    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 00:47:22.750047 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.749894    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f27qr\""
Jul 28 00:47:22.750047 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.749932    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 00:47:22.750047 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.749978    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkfth\""
Jul 28 00:47:22.750047 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.749991    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 00:47:22.756538 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756518    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695d7f7-26eb-4dce-82f7-7b3020213bed-data-volume\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.756652 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756555    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7089b41c-0d46-4691-bb98-b3f46dd0887f-cert\") pod \"ingress-canary-mlmxc\" (UID: \"7089b41c-0d46-4691-bb98-b3f46dd0887f\") " pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.756652 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7adf268e-0c53-4229-8416-847cffcc6dc2-metrics-tls\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.756652 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a695d7f7-26eb-4dce-82f7-7b3020213bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.756652 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9fg\" (UniqueName: \"kubernetes.io/projected/a695d7f7-26eb-4dce-82f7-7b3020213bed-kube-api-access-vj9fg\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.756874 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a695d7f7-26eb-4dce-82f7-7b3020213bed-crio-socket\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.756874 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756777    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a695d7f7-26eb-4dce-82f7-7b3020213bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.756874 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7adf268e-0c53-4229-8416-847cffcc6dc2-config-volume\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.757042 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adf268e-0c53-4229-8416-847cffcc6dc2-tmp-dir\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.757042 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756926    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/7089b41c-0d46-4691-bb98-b3f46dd0887f-kube-api-access-4djkk\") pod \"ingress-canary-mlmxc\" (UID: \"7089b41c-0d46-4691-bb98-b3f46dd0887f\") " pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.757042 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.756956    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbqfk\" (UniqueName: \"kubernetes.io/projected/7adf268e-0c53-4229-8416-847cffcc6dc2-kube-api-access-bbqfk\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.836227 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.836204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:22.857354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a695d7f7-26eb-4dce-82f7-7b3020213bed-crio-socket\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.857418 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a695d7f7-26eb-4dce-82f7-7b3020213bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.857418 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7adf268e-0c53-4229-8416-847cffcc6dc2-config-volume\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.857493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adf268e-0c53-4229-8416-847cffcc6dc2-tmp-dir\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.857493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/7089b41c-0d46-4691-bb98-b3f46dd0887f-kube-api-access-4djkk\") pod \"ingress-canary-mlmxc\" (UID: \"7089b41c-0d46-4691-bb98-b3f46dd0887f\") " pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.857493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbqfk\" (UniqueName: \"kubernetes.io/projected/7adf268e-0c53-4229-8416-847cffcc6dc2-kube-api-access-bbqfk\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.857493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695d7f7-26eb-4dce-82f7-7b3020213bed-data-volume\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.857675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7089b41c-0d46-4691-bb98-b3f46dd0887f-cert\") pod \"ingress-canary-mlmxc\" (UID: \"7089b41c-0d46-4691-bb98-b3f46dd0887f\") " pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.857675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7adf268e-0c53-4229-8416-847cffcc6dc2-metrics-tls\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.857675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857558    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a695d7f7-26eb-4dce-82f7-7b3020213bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.857675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9fg\" (UniqueName: \"kubernetes.io/projected/a695d7f7-26eb-4dce-82f7-7b3020213bed-kube-api-access-vj9fg\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.857675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857583    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a695d7f7-26eb-4dce-82f7-7b3020213bed-crio-socket\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.857945 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857782    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7adf268e-0c53-4229-8416-847cffcc6dc2-tmp-dir\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.857945 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.857840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a695d7f7-26eb-4dce-82f7-7b3020213bed-data-volume\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.858058 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.858002    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a695d7f7-26eb-4dce-82f7-7b3020213bed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.858303 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.858285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7adf268e-0c53-4229-8416-847cffcc6dc2-config-volume\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.861303 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.861276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7adf268e-0c53-4229-8416-847cffcc6dc2-metrics-tls\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.861419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.861398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a695d7f7-26eb-4dce-82f7-7b3020213bed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.861502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.861489    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7089b41c-0d46-4691-bb98-b3f46dd0887f-cert\") pod \"ingress-canary-mlmxc\" (UID: \"7089b41c-0d46-4691-bb98-b3f46dd0887f\") " pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.865403 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.865384    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbqfk\" (UniqueName: \"kubernetes.io/projected/7adf268e-0c53-4229-8416-847cffcc6dc2-kube-api-access-bbqfk\") pod \"dns-default-2nsh7\" (UID: \"7adf268e-0c53-4229-8416-847cffcc6dc2\") " pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:22.865901 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.865879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9fg\" (UniqueName: \"kubernetes.io/projected/a695d7f7-26eb-4dce-82f7-7b3020213bed-kube-api-access-vj9fg\") pod \"insights-runtime-extractor-bzfc2\" (UID: \"a695d7f7-26eb-4dce-82f7-7b3020213bed\") " pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:22.865985 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.865925    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4djkk\" (UniqueName: \"kubernetes.io/projected/7089b41c-0d46-4691-bb98-b3f46dd0887f-kube-api-access-4djkk\") pod \"ingress-canary-mlmxc\" (UID: \"7089b41c-0d46-4691-bb98-b3f46dd0887f\") " pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:22.896207 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:22.896143    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:23.014834 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.014807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:23.035090 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.035072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mlmxc"
Jul 28 00:47:23.054567 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.054546    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-bzfc2"
Jul 28 00:47:23.241657 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.241485    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mlmxc"]
Jul 28 00:47:23.245631 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:23.245601    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7089b41c_0d46_4691_bb98_b3f46dd0887f.slice/crio-611421a073cbb32fd76f24f2326ad5dbff6e8ed591ea1829284225228b4d14cf WatchSource:0}: Error finding container 611421a073cbb32fd76f24f2326ad5dbff6e8ed591ea1829284225228b4d14cf: Status 404 returned error can't find the container with id 611421a073cbb32fd76f24f2326ad5dbff6e8ed591ea1829284225228b4d14cf
Jul 28 00:47:23.254141 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.254118    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-2nsh7"]
Jul 28 00:47:23.257371 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.257343    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-bzfc2"]
Jul 28 00:47:23.257783 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:23.257674    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adf268e_0c53_4229_8416_847cffcc6dc2.slice/crio-309dce87cbfb940017666c4f1c4f7c44bca76fd8794948c5caf0bcdfd341f33c WatchSource:0}: Error finding container 309dce87cbfb940017666c4f1c4f7c44bca76fd8794948c5caf0bcdfd341f33c: Status 404 returned error can't find the container with id 309dce87cbfb940017666c4f1c4f7c44bca76fd8794948c5caf0bcdfd341f33c
Jul 28 00:47:23.259668 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:23.259645    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda695d7f7_26eb_4dce_82f7_7b3020213bed.slice/crio-b9bc1250b9bad53406500e71b7b3ed6b49aaddeb907ce64538360fd614236466 WatchSource:0}: Error finding container b9bc1250b9bad53406500e71b7b3ed6b49aaddeb907ce64538360fd614236466: Status 404 returned error can't find the container with id b9bc1250b9bad53406500e71b7b3ed6b49aaddeb907ce64538360fd614236466
Jul 28 00:47:23.749111 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.749077    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:23.753856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.753723    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 00:47:23.753856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.753733    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kdcl9\""
Jul 28 00:47:23.878235 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.878201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:23.980017 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:23.979979    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:24.054243 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.054154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzfc2" event={"ID":"a695d7f7-26eb-4dce-82f7-7b3020213bed","Type":"ContainerStarted","Data":"b9bc1250b9bad53406500e71b7b3ed6b49aaddeb907ce64538360fd614236466"}
Jul 28 00:47:24.055886 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.055856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nsh7" event={"ID":"7adf268e-0c53-4229-8416-847cffcc6dc2","Type":"ContainerStarted","Data":"309dce87cbfb940017666c4f1c4f7c44bca76fd8794948c5caf0bcdfd341f33c"}
Jul 28 00:47:24.057339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.057309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mlmxc" event={"ID":"7089b41c-0d46-4691-bb98-b3f46dd0887f","Type":"ContainerStarted","Data":"611421a073cbb32fd76f24f2326ad5dbff6e8ed591ea1829284225228b4d14cf"}
Jul 28 00:47:24.321955 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.321897    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:24.413561 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.413524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:24.570542 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.570470    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"]
Jul 28 00:47:24.593676 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.593584    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"]
Jul 28 00:47:24.593676 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.593638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:24.594390 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.593970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:24.597834 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.597665    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 28 00:47:24.598136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.597976    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gmndh\""
Jul 28 00:47:24.652790 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.652713    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:24.673545 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.673509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e0fa5-39b3-4c7c-89d6-b35e5d703969-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-v5zmn\" (UID: \"6f4e0fa5-39b3-4c7c-89d6-b35e5d703969\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:24.753301 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.753265    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:24.774438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:24.774385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e0fa5-39b3-4c7c-89d6-b35e5d703969-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-v5zmn\" (UID: \"6f4e0fa5-39b3-4c7c-89d6-b35e5d703969\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:24.774663 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:24.774553    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 00:47:24.774663 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:24.774624    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f4e0fa5-39b3-4c7c-89d6-b35e5d703969-tls-certificates podName:6f4e0fa5-39b3-4c7c-89d6-b35e5d703969 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:25.274603702 +0000 UTC m=+50.023742410 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/6f4e0fa5-39b3-4c7c-89d6-b35e5d703969-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-v5zmn" (UID: "6f4e0fa5-39b3-4c7c-89d6-b35e5d703969") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 28 00:47:25.169429 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.169394    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:25.247762 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.247730    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8db66f99c-95smh"]
Jul 28 00:47:25.261945 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.261911    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8db66f99c-95smh"]
Jul 28 00:47:25.262145 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.262006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.265028 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.265008    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 00:47:25.266237 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266200    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 00:47:25.266430 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266261    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 00:47:25.266430 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266271    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 00:47:25.266547 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266459    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-xdbv6\""
Jul 28 00:47:25.266547 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 00:47:25.266649 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266558    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 00:47:25.266797 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.266771    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 00:47:25.279132 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-oauth-serving-cert\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.279241 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-oauth-config\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.279241 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-service-ca\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.279364 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-console-config\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.279364 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e0fa5-39b3-4c7c-89d6-b35e5d703969-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-v5zmn\" (UID: \"6f4e0fa5-39b3-4c7c-89d6-b35e5d703969\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:25.279364 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279347    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-serving-cert\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.279507 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.279374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88f7n\" (UniqueName: \"kubernetes.io/projected/82d3863f-96e7-4145-a4b6-d4c474e48261-kube-api-access-88f7n\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.283604 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.283582    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6f4e0fa5-39b3-4c7c-89d6-b35e5d703969-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-v5zmn\" (UID: \"6f4e0fa5-39b3-4c7c-89d6-b35e5d703969\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:25.380495 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.380465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-oauth-serving-cert\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.380684 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.380504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-oauth-config\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.380684 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.380529    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-service-ca\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.380684 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.380554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-console-config\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.380684 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.380605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-serving-cert\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.380684 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.380631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88f7n\" (UniqueName: \"kubernetes.io/projected/82d3863f-96e7-4145-a4b6-d4c474e48261-kube-api-access-88f7n\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.381314 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.381287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-oauth-serving-cert\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.381446 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.381376    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-service-ca\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.381508 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.381459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-console-config\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.383559 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.383539    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-oauth-config\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.383702 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.383679    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-serving-cert\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.389837 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.389792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88f7n\" (UniqueName: \"kubernetes.io/projected/82d3863f-96e7-4145-a4b6-d4c474e48261-kube-api-access-88f7n\") pod \"console-8db66f99c-95smh\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") " pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.507524 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.507437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:25.573624 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.573421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:25.758869 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.758793    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:25.864756 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:25.864630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:29.811617 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:29.811340    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8db66f99c-95smh"]
Jul 28 00:47:29.817094 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:29.817036    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"]
Jul 28 00:47:30.094921 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:30.094889    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d3863f_96e7_4145_a4b6_d4c474e48261.slice/crio-72171cdaca3918c04004091de9a050b22b9ccae16dd2ee01d678f34a988e35e6 WatchSource:0}: Error finding container 72171cdaca3918c04004091de9a050b22b9ccae16dd2ee01d678f34a988e35e6: Status 404 returned error can't find the container with id 72171cdaca3918c04004091de9a050b22b9ccae16dd2ee01d678f34a988e35e6
Jul 28 00:47:30.095666 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:30.095637    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f4e0fa5_39b3_4c7c_89d6_b35e5d703969.slice/crio-d785759b39fba39f170d1733f547d4aa2f5f8b14e4c0e4cdb7f1684e6687b0ff WatchSource:0}: Error finding container d785759b39fba39f170d1733f547d4aa2f5f8b14e4c0e4cdb7f1684e6687b0ff: Status 404 returned error can't find the container with id d785759b39fba39f170d1733f547d4aa2f5f8b14e4c0e4cdb7f1684e6687b0ff
Jul 28 00:47:30.623570 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:30.623445    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:30.674723 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:30.674686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:31.082485 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.082448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nsh7" event={"ID":"7adf268e-0c53-4229-8416-847cffcc6dc2","Type":"ContainerStarted","Data":"3dc8ae2f58985a81df748a2ad8edc38d7fe76d626be5aae7ca0a8fa1875be2c2"}
Jul 28 00:47:31.083154 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.082559    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-2nsh7" event={"ID":"7adf268e-0c53-4229-8416-847cffcc6dc2","Type":"ContainerStarted","Data":"87b83008be1f3962cb16b21c002c024973ee9e7b153b3ae09e7b0cde35f25a03"}
Jul 28 00:47:31.083412 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.083391    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:31.086596 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.086568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn" event={"ID":"6f4e0fa5-39b3-4c7c-89d6-b35e5d703969","Type":"ContainerStarted","Data":"d785759b39fba39f170d1733f547d4aa2f5f8b14e4c0e4cdb7f1684e6687b0ff"}
Jul 28 00:47:31.088990 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.088966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8db66f99c-95smh" event={"ID":"82d3863f-96e7-4145-a4b6-d4c474e48261","Type":"ContainerStarted","Data":"72171cdaca3918c04004091de9a050b22b9ccae16dd2ee01d678f34a988e35e6"}
Jul 28 00:47:31.091426 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.091401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzfc2" event={"ID":"a695d7f7-26eb-4dce-82f7-7b3020213bed","Type":"ContainerStarted","Data":"013564a0cf45c487d41e0641195f63e9a620231c2c4dd06084c4aa51ce201254"}
Jul 28 00:47:31.097176 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.096706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mlmxc" event={"ID":"7089b41c-0d46-4691-bb98-b3f46dd0887f","Type":"ContainerStarted","Data":"1da3bf8de46ad9bf09653ce581a170090b00e00a543d81476c678852b6742be7"}
Jul 28 00:47:31.101855 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.101442    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-2nsh7" podStartSLOduration=2.267988764 podStartE2EDuration="9.10142547s" totalImagesPullingTime="6.833436706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:31.10029546 +0000 UTC m=+55.849434180" watchObservedRunningTime="2026-07-28 00:47:31.10142547 +0000 UTC m=+55.850564197"
Jul 28 00:47:31.150348 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.150202    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mlmxc" podStartSLOduration=2.304827611 podStartE2EDuration="9.150185469s" totalImagesPullingTime="6.845357858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:31.14994668 +0000 UTC m=+55.899085405" watchObservedRunningTime="2026-07-28 00:47:31.150185469 +0000 UTC m=+55.899324194"
Jul 28 00:47:31.967857 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:31.967585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:32.014916 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.014881    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:32.102109 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.102034    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn" event={"ID":"6f4e0fa5-39b3-4c7c-89d6-b35e5d703969","Type":"ContainerStarted","Data":"2c9ff8c8258284df0190136f0b66babd05bb1aa07f88e40b329690470e265830"}
Jul 28 00:47:32.102605 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.102570    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:32.108766 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.108741    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn"
Jul 28 00:47:32.118782 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.118734    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-v5zmn" podStartSLOduration=6.717782536 podStartE2EDuration="8.118717675s" totalImagesPullingTime="1.400935139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:32.11751359 +0000 UTC m=+56.866652314" watchObservedRunningTime="2026-07-28 00:47:32.118717675 +0000 UTC m=+56.867856395"
Jul 28 00:47:32.593049 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.593017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:32.692370 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:32.692334    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:33.215736 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.215700    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b74b69b8f-rpk8b"]
Jul 28 00:47:33.223779 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.223757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.230946 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.230845    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b74b69b8f-rpk8b"]
Jul 28 00:47:33.232983 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.232962    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 00:47:33.249039 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-config\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.249151 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249042    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-oauth-serving-cert\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.249151 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249061    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-serving-cert\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.249151 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvvhw\" (UniqueName: \"kubernetes.io/projected/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-kube-api-access-vvvhw\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.249339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-oauth-config\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.249339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-service-ca\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.249339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.249313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-trusted-ca-bundle\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.251317 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.251295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:33.349821 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.349789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-service-ca\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.349821 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.349825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-trusted-ca-bundle\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350046 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.349861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-config\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350101 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-oauth-serving-cert\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350148 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-serving-cert\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350183 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350158    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvvhw\" (UniqueName: \"kubernetes.io/projected/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-kube-api-access-vvvhw\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350976 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350684    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-service-ca\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350976 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-config\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350976 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350787    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-trusted-ca-bundle\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.350976 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-oauth-serving-cert\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.351268 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.350991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-oauth-config\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.355333 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.355310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-oauth-config\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.355453 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.355432    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-serving-cert\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.359327 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.359306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvvhw\" (UniqueName: \"kubernetes.io/projected/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-kube-api-access-vvvhw\") pod \"console-6b74b69b8f-rpk8b\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") " pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.534908 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.534829    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:33.831295 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.831261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:33.934410 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:33.934380    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:35.310713 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:35.310670    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b74b69b8f-rpk8b"]
Jul 28 00:47:35.314013 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:35.313983    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f6a7ae_6f6b_4f5f_86ec_1bc1919f7cc4.slice/crio-2d3ce682dfc2e3ee89f8846116bed36366425a2e2648dc49ecff2696b710dc9f WatchSource:0}: Error finding container 2d3ce682dfc2e3ee89f8846116bed36366425a2e2648dc49ecff2696b710dc9f: Status 404 returned error can't find the container with id 2d3ce682dfc2e3ee89f8846116bed36366425a2e2648dc49ecff2696b710dc9f
Jul 28 00:47:35.605164 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:35.605126    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:36.013393 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.013321    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-j849f"
Jul 28 00:47:36.116428 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.116387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzfc2" event={"ID":"a695d7f7-26eb-4dce-82f7-7b3020213bed","Type":"ContainerStarted","Data":"0f627ee66e58f75c4929c93d8b6d19d23fd813ba0c4a9217eb377df9562bb978"}
Jul 28 00:47:36.118017 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.117988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8db66f99c-95smh" event={"ID":"82d3863f-96e7-4145-a4b6-d4c474e48261","Type":"ContainerStarted","Data":"af750b9970e34ad4d2743e45e57e39495b0edacd77b462e2acd448fa01569105"}
Jul 28 00:47:36.119846 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.119816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74b69b8f-rpk8b" event={"ID":"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4","Type":"ContainerStarted","Data":"cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb"}
Jul 28 00:47:36.119977 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.119857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74b69b8f-rpk8b" event={"ID":"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4","Type":"ContainerStarted","Data":"2d3ce682dfc2e3ee89f8846116bed36366425a2e2648dc49ecff2696b710dc9f"}
Jul 28 00:47:36.134048 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.134003    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8db66f99c-95smh" podStartSLOduration=6.044797746 podStartE2EDuration="11.133987383s" totalImagesPullingTime="5.089189637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:36.133270452 +0000 UTC m=+60.882409175" watchObservedRunningTime="2026-07-28 00:47:36.133987383 +0000 UTC m=+60.883126108"
Jul 28 00:47:36.139025 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.138996    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:36.150673 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.150635    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b74b69b8f-rpk8b" podStartSLOduration=3.150618659 podStartE2EDuration="3.150618659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:36.149250149 +0000 UTC m=+60.898388877" watchObservedRunningTime="2026-07-28 00:47:36.150618659 +0000 UTC m=+60.899757385"
Jul 28 00:47:36.201319 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:36.201295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:37.573370 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:37.573334    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:37.674745 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:37.674710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:37.773400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:37.773369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:37.959675 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:37.959601    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:37.990439 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:37.990407    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"]
Jul 28 00:47:38.006184 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.006159    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"]
Jul 28 00:47:38.006330 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.006302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.009386 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.009362    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 00:47:38.009386 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.009384    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 00:47:38.009560 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.009470    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 00:47:38.009560 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.009531    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 00:47:38.009744 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.009726    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-jrqst\""
Jul 28 00:47:38.011339 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.011297    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 00:47:38.015775 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.015406    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"]
Jul 28 00:47:38.019042 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.019024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.022028 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.022005    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-684xl\""
Jul 28 00:47:38.022125 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.022043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 00:47:38.022125 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.022061    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 00:47:38.022125 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.022085    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 00:47:38.022348 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.022333    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sqgsp"]
Jul 28 00:47:38.026933 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.026908    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.030763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.030338    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 00:47:38.030763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.030437    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hpvwn\""
Jul 28 00:47:38.031459 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.030973    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 00:47:38.031459 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.031037    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 00:47:38.037541 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.035113    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"]
Jul 28 00:47:38.054181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.054162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.091323 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6dc160e0-e027-460b-b5a0-ccd194d013a6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.091441 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091345    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.091441 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5020ccf5-e631-4e00-8445-11a51559a7e4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.091441 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f59960c-15ec-4904-ade3-e51e6e503ede-metrics-client-ca\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.091607 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091481    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.091607 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091546    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8wg\" (UniqueName: \"kubernetes.io/projected/5020ccf5-e631-4e00-8445-11a51559a7e4-kube-api-access-2p8wg\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.091607 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091580    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.091763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.091763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc160e0-e027-460b-b5a0-ccd194d013a6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.091763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091706    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54h4v\" (UniqueName: \"kubernetes.io/projected/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-api-access-54h4v\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.091763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-root\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.091763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091745    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091790    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-sys\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-wtmp\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-textfile\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-tls\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.092022 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.091927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppqx\" (UniqueName: \"kubernetes.io/projected/0f59960c-15ec-4904-ade3-e51e6e503ede-kube-api-access-5ppqx\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.116357 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.116333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.192595 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-wtmp\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.192595 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192596    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.192786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-textfile\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.192786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-tls\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.192786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192685    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppqx\" (UniqueName: \"kubernetes.io/projected/0f59960c-15ec-4904-ade3-e51e6e503ede-kube-api-access-5ppqx\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.192786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6dc160e0-e027-460b-b5a0-ccd194d013a6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.192786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-wtmp\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.192786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5020ccf5-e631-4e00-8445-11a51559a7e4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f59960c-15ec-4904-ade3-e51e6e503ede-metrics-client-ca\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192856    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8wg\" (UniqueName: \"kubernetes.io/projected/5020ccf5-e631-4e00-8445-11a51559a7e4-kube-api-access-2p8wg\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.192980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.193001    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc160e0-e027-460b-b5a0-ccd194d013a6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.193017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54h4v\" (UniqueName: \"kubernetes.io/projected/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-api-access-54h4v\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.193024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-textfile\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.193036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-root\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.193095 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.193062    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.193977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5020ccf5-e631-4e00-8445-11a51559a7e4-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6dc160e0-e027-460b-b5a0-ccd194d013a6-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-sys\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-sys\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f59960c-15ec-4904-ade3-e51e6e503ede-root\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:38.194314    2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:47:38.194400    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-tls podName:5020ccf5-e631-4e00-8445-11a51559a7e4 nodeName:}" failed. No retries permitted until 2026-07-28 00:47:38.694380678 +0000 UTC m=+63.443519385 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-mchrn" (UID: "5020ccf5-e631-4e00-8445-11a51559a7e4") : secret "openshift-state-metrics-tls" not found
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194680    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f59960c-15ec-4904-ade3-e51e6e503ede-metrics-client-ca\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.194999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.194953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-accelerators-collector-config\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.195541 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.195287    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dc160e0-e027-460b-b5a0-ccd194d013a6-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.195541 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.195506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-tls\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.195885 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.195817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f59960c-15ec-4904-ade3-e51e6e503ede-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.196020 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.195997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.196733 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.196711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.196821 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.196758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.202988 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.202969    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8wg\" (UniqueName: \"kubernetes.io/projected/5020ccf5-e631-4e00-8445-11a51559a7e4-kube-api-access-2p8wg\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.203091 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.203075    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppqx\" (UniqueName: \"kubernetes.io/projected/0f59960c-15ec-4904-ade3-e51e6e503ede-kube-api-access-5ppqx\") pod \"node-exporter-sqgsp\" (UID: \"0f59960c-15ec-4904-ade3-e51e6e503ede\") " pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.203751 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.203727    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54h4v\" (UniqueName: \"kubernetes.io/projected/6dc160e0-e027-460b-b5a0-ccd194d013a6-kube-api-access-54h4v\") pod \"kube-state-metrics-56d9f944b4-vvhtb\" (UID: \"6dc160e0-e027-460b-b5a0-ccd194d013a6\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.230918 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.230828    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.329343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.329289    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"
Jul 28 00:47:38.329518 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.329403    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.343989 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.343968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sqgsp"
Jul 28 00:47:38.432371 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.432340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.587697 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.587664    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.688612 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.688583    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:38.698765 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.698736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.701070 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.701045    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/5020ccf5-e631-4e00-8445-11a51559a7e4-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mchrn\" (UID: \"5020ccf5-e631-4e00-8445-11a51559a7e4\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.704163 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:38.704134    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f59960c_15ec_4904_ade3_e51e6e503ede.slice/crio-40bc383d82ccf55e3b58d7a76e9c1080687075c908f5f39f620ee4d1a2b37f96 WatchSource:0}: Error finding container 40bc383d82ccf55e3b58d7a76e9c1080687075c908f5f39f620ee4d1a2b37f96: Status 404 returned error can't find the container with id 40bc383d82ccf55e3b58d7a76e9c1080687075c908f5f39f620ee4d1a2b37f96
Jul 28 00:47:38.822583 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.822552    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb"]
Jul 28 00:47:38.824933 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:38.824910    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc160e0_e027_460b_b5a0_ccd194d013a6.slice/crio-84a09e1e29d008545dfcf36b9b56f6250dff250b0b928c93b32d9a138b0359b6 WatchSource:0}: Error finding container 84a09e1e29d008545dfcf36b9b56f6250dff250b0b928c93b32d9a138b0359b6: Status 404 returned error can't find the container with id 84a09e1e29d008545dfcf36b9b56f6250dff250b0b928c93b32d9a138b0359b6
Jul 28 00:47:38.918438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.918388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"
Jul 28 00:47:38.972886 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:38.972858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:39.040332 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.040278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn"]
Jul 28 00:47:39.042738 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:39.042710    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5020ccf5_e631_4e00_8445_11a51559a7e4.slice/crio-6bc4f4809353ff8b8b2e4b2e4a95739a08e7cc7254daa8a001c67a49ec0093fa WatchSource:0}: Error finding container 6bc4f4809353ff8b8b2e4b2e4a95739a08e7cc7254daa8a001c67a49ec0093fa: Status 404 returned error can't find the container with id 6bc4f4809353ff8b8b2e4b2e4a95739a08e7cc7254daa8a001c67a49ec0093fa
Jul 28 00:47:39.056351 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.056328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:39.061490 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.061469    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 00:47:39.065836 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.065817    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.068579 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068556    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 00:47:39.068676 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068618    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 00:47:39.068676 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068618    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 00:47:39.068936 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068916    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 00:47:39.069000 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068959    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 00:47:39.069000 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068993    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-hnbvj\""
Jul 28 00:47:39.069097 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068961    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 00:47:39.069097 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068964    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 00:47:39.069097 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.068924    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 00:47:39.069239 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.069176    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 00:47:39.080903 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.080873    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 00:47:39.133104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.133076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqgsp" event={"ID":"0f59960c-15ec-4904-ade3-e51e6e503ede","Type":"ContainerStarted","Data":"40bc383d82ccf55e3b58d7a76e9c1080687075c908f5f39f620ee4d1a2b37f96"}
Jul 28 00:47:39.136039 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.136009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-bzfc2" event={"ID":"a695d7f7-26eb-4dce-82f7-7b3020213bed","Type":"ContainerStarted","Data":"71816e77db0e7e61b912afd8544089984c38b5a834b9b00ffab36cca17dd2920"}
Jul 28 00:47:39.138116 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.138055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb" event={"ID":"6dc160e0-e027-460b-b5a0-ccd194d013a6","Type":"ContainerStarted","Data":"84a09e1e29d008545dfcf36b9b56f6250dff250b0b928c93b32d9a138b0359b6"}
Jul 28 00:47:39.140482 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.140461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn" event={"ID":"5020ccf5-e631-4e00-8445-11a51559a7e4","Type":"ContainerStarted","Data":"a0d6ed8dbb83045c105870bccd1c37d7daa6077b8fe13b9d5363ce5143bff1a4"}
Jul 28 00:47:39.140593 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.140513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn" event={"ID":"5020ccf5-e631-4e00-8445-11a51559a7e4","Type":"ContainerStarted","Data":"6bc4f4809353ff8b8b2e4b2e4a95739a08e7cc7254daa8a001c67a49ec0093fa"}
Jul 28 00:47:39.153567 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.153531    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-bzfc2" podStartSLOduration=1.857603411 podStartE2EDuration="17.15351657s" totalImagesPullingTime="15.295913159s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:47:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:39.152512527 +0000 UTC m=+63.901651253" watchObservedRunningTime="2026-07-28 00:47:39.15351657 +0000 UTC m=+63.902655330"
Jul 28 00:47:39.164584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.164560    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:39.202635 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.202607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-out\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.202722 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.202652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.202759 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.202740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.202790 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.202776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9w5\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-kube-api-access-hj9w5\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.202830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.202801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203137 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.202957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203137 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203137 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203137 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203074    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203137 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-web-config\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203439 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203158    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203439 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203210    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.203439 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.203262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.278438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.278393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:39.304139 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-web-config\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304481 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304481 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304481 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304494    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-out\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9w5\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-kube-api-access-hj9w5\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304674    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304958 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304709    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.304958 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.304890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.306626 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.306567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.306910 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.306884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.307254 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.307197    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-web-config\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.307348 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.307294    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.307634 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.307591    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.308272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.307779    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.308272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.308193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.309320 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.309266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.309487 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.309453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.309735 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.309707    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.310031 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.310011    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-out\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.312990 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.312972    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9w5\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-kube-api-access-hj9w5\") pod \"alertmanager-main-0\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.378786 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.378745    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:39.379101 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.379082    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:47:39.693122 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:39.693001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 00:47:39.696856 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:39.696826    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6990f9d_1c41_48b7_8c8f_e4db42135f02.slice/crio-3d18c7791fd3ff874c1d9627d7ebb2365f9903433740a0ddd2bee60dbb556954 WatchSource:0}: Error finding container 3d18c7791fd3ff874c1d9627d7ebb2365f9903433740a0ddd2bee60dbb556954: Status 404 returned error can't find the container with id 3d18c7791fd3ff874c1d9627d7ebb2365f9903433740a0ddd2bee60dbb556954
Jul 28 00:47:40.030963 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.030933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:40.145811 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.145773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"3d18c7791fd3ff874c1d9627d7ebb2365f9903433740a0ddd2bee60dbb556954"}
Jul 28 00:47:40.147582 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.147550    2576 generic.go:356] "Generic (PLEG): container finished" podID="0f59960c-15ec-4904-ade3-e51e6e503ede" containerID="795b7cbc868011c4592be096e294603609c9e3892e192934c35d2283cf38e2ef" exitCode=0
Jul 28 00:47:40.147710 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.147636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqgsp" event={"ID":"0f59960c-15ec-4904-ade3-e51e6e503ede","Type":"ContainerDied","Data":"795b7cbc868011c4592be096e294603609c9e3892e192934c35d2283cf38e2ef"}
Jul 28 00:47:40.149824 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.149804    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn" event={"ID":"5020ccf5-e631-4e00-8445-11a51559a7e4","Type":"ContainerStarted","Data":"4fccb266a778e2c4e258fd511c77dd08c5ed27713e8bcb2f9bfa0dffcd7d4dfe"}
Jul 28 00:47:40.923060 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.923018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:40.984419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:40.984379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:41.071444 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.071416    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:41.155820 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.155792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb" event={"ID":"6dc160e0-e027-460b-b5a0-ccd194d013a6","Type":"ContainerStarted","Data":"709b05259ebf296ffc9fa865e9da5de3481183f1c28d27384c605d6bc66d8297"}
Jul 28 00:47:41.155921 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.155844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb" event={"ID":"6dc160e0-e027-460b-b5a0-ccd194d013a6","Type":"ContainerStarted","Data":"d19df0e455c0b8b7ae41c6a5ab53c06390174827d7a4bc93b3d078b084b2c02f"}
Jul 28 00:47:41.155921 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.155868    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb" event={"ID":"6dc160e0-e027-460b-b5a0-ccd194d013a6","Type":"ContainerStarted","Data":"05164d31404186347d08194b0d5f805dee87ec53bef3f64c0e244e6224b67f04"}
Jul 28 00:47:41.157800 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.157781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn" event={"ID":"5020ccf5-e631-4e00-8445-11a51559a7e4","Type":"ContainerStarted","Data":"981faa6c9cbb783d255822098b0f054e80d78ce254dd2b34bcdf5b795f19221f"}
Jul 28 00:47:41.159278 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.159259    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99" exitCode=0
Jul 28 00:47:41.159356 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.159335    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"}
Jul 28 00:47:41.161404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.161387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqgsp" event={"ID":"0f59960c-15ec-4904-ade3-e51e6e503ede","Type":"ContainerStarted","Data":"127c627dc0f8ea7c295bb4ba713433e6197b89bc2127705852d18dc0a54ab48a"}
Jul 28 00:47:41.161468 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.161422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sqgsp" event={"ID":"0f59960c-15ec-4904-ade3-e51e6e503ede","Type":"ContainerStarted","Data":"007c380915124dd8d38077899621ec5ac279771e0f74b75eafced16c70605cf6"}
Jul 28 00:47:41.176321 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.176296    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:41.177073 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.177039    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-vvhtb" podStartSLOduration=2.628394976 podStartE2EDuration="4.177026262s" totalImagesPullingTime="1.548631286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:41.176639845 +0000 UTC m=+65.925778568" watchObservedRunningTime="2026-07-28 00:47:41.177026262 +0000 UTC m=+65.926164985"
Jul 28 00:47:41.193906 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.193873    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mchrn" podStartSLOduration=2.989182046 podStartE2EDuration="4.193862144s" totalImagesPullingTime="1.204680098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:41.193316132 +0000 UTC m=+65.942454857" watchObservedRunningTime="2026-07-28 00:47:41.193862144 +0000 UTC m=+65.943000868"
Jul 28 00:47:41.237889 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.237855    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sqgsp" podStartSLOduration=2.268082456 podStartE2EDuration="3.237844169s" totalImagesPullingTime="891.411515ms" totalInitContainerRuntime="78.350198ms" isStatefulPod=true podCreationTimestamp="2026-07-28 00:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:41.237162673 +0000 UTC m=+65.986301398" watchObservedRunningTime="2026-07-28 00:47:41.237844169 +0000 UTC m=+65.986982892"
Jul 28 00:47:41.422160 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.422083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:41.425031 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.425013    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 00:47:41.435202 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.435176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5cd31cd7-5079-487f-9a62-06a8854a3f58-original-pull-secret\") pod \"global-pull-secret-syncer-zw59q\" (UID: \"5cd31cd7-5079-487f-9a62-06a8854a3f58\") " pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:41.522467 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.522428    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:41.525231 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.525206    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 00:47:41.534735 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.534711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c50f947-74a7-4285-adaa-0e77f24fc849-metrics-certs\") pod \"network-metrics-daemon-f4x8d\" (UID: \"3c50f947-74a7-4285-adaa-0e77f24fc849\") " pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:41.623857 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.623830    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:41.626499 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.626482    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 00:47:41.639878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.639862    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 00:47:41.647015 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.646993    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfkt\" (UniqueName: \"kubernetes.io/projected/8e2d0561-a486-4f89-a50e-9f9e5cbfe537-kube-api-access-ncfkt\") pod \"network-check-target-l6shq\" (UID: \"8e2d0561-a486-4f89-a50e-9f9e5cbfe537\") " pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:41.664306 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.664290    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mkfth\""
Jul 28 00:47:41.665304 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.665290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zw59q"
Jul 28 00:47:41.672148 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.672128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:41.763852 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.763828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kdcl9\""
Jul 28 00:47:41.771668 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.771495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-f4x8d"
Jul 28 00:47:41.797524 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.797497    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zw59q"]
Jul 28 00:47:41.801320 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:41.801294    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd31cd7_5079_487f_9a62_06a8854a3f58.slice/crio-5c54994f5ab2caccf02a63bb3db46eccf144806b30cbb3b2ccca381062a938cf WatchSource:0}: Error finding container 5c54994f5ab2caccf02a63bb3db46eccf144806b30cbb3b2ccca381062a938cf: Status 404 returned error can't find the container with id 5c54994f5ab2caccf02a63bb3db46eccf144806b30cbb3b2ccca381062a938cf
Jul 28 00:47:41.811390 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.811370    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-l6shq"]
Jul 28 00:47:41.813818 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:41.813795    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2d0561_a486_4f89_a50e_9f9e5cbfe537.slice/crio-1ff0117667b85d1dfc04f95d5d695536ed4a77ce00f8d094ca29e09e820f0db3 WatchSource:0}: Error finding container 1ff0117667b85d1dfc04f95d5d695536ed4a77ce00f8d094ca29e09e820f0db3: Status 404 returned error can't find the container with id 1ff0117667b85d1dfc04f95d5d695536ed4a77ce00f8d094ca29e09e820f0db3
Jul 28 00:47:41.906726 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:41.906695    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-f4x8d"]
Jul 28 00:47:41.909355 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:41.909327    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c50f947_74a7_4285_adaa_0e77f24fc849.slice/crio-bc19dce162f3084e4ba4a0766da2de3e6b4520d638d008f7806a40ef6913fa36 WatchSource:0}: Error finding container bc19dce162f3084e4ba4a0766da2de3e6b4520d638d008f7806a40ef6913fa36: Status 404 returned error can't find the container with id bc19dce162f3084e4ba4a0766da2de3e6b4520d638d008f7806a40ef6913fa36
Jul 28 00:47:42.107324 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.107266    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-2nsh7"
Jul 28 00:47:42.167210 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.167175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zw59q" event={"ID":"5cd31cd7-5079-487f-9a62-06a8854a3f58","Type":"ContainerStarted","Data":"5c54994f5ab2caccf02a63bb3db46eccf144806b30cbb3b2ccca381062a938cf"}
Jul 28 00:47:42.168525 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.168495    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6shq" event={"ID":"8e2d0561-a486-4f89-a50e-9f9e5cbfe537","Type":"ContainerStarted","Data":"1ff0117667b85d1dfc04f95d5d695536ed4a77ce00f8d094ca29e09e820f0db3"}
Jul 28 00:47:42.169900 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.169874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f4x8d" event={"ID":"3c50f947-74a7-4285-adaa-0e77f24fc849","Type":"ContainerStarted","Data":"bc19dce162f3084e4ba4a0766da2de3e6b4520d638d008f7806a40ef6913fa36"}
Jul 28 00:47:42.312371 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.312338    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-ff65875f-tlftv"]
Jul 28 00:47:42.315732 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.315706    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.318986 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.318964    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 00:47:42.319305 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.319286    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fe30oa2p5d3rh\""
Jul 28 00:47:42.319528 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.319508    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-79gxs\""
Jul 28 00:47:42.319857 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.319835    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 00:47:42.320014 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.319994    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 00:47:42.320096 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.320007    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 00:47:42.324711 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.324646    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-ff65875f-tlftv"]
Jul 28 00:47:42.431200 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-client-ca-bundle\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.431200 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad58434d-62ed-4820-bd0b-71f91dfd24e6-metrics-server-audit-profiles\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.431432 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431261    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84lw9\" (UniqueName: \"kubernetes.io/projected/ad58434d-62ed-4820-bd0b-71f91dfd24e6-kube-api-access-84lw9\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.431432 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-secret-metrics-server-tls\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.431432 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-secret-metrics-server-client-certs\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.431432 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad58434d-62ed-4820-bd0b-71f91dfd24e6-audit-log\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.431432 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.431403    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad58434d-62ed-4820-bd0b-71f91dfd24e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad58434d-62ed-4820-bd0b-71f91dfd24e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532449 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-client-ca-bundle\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532449 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad58434d-62ed-4820-bd0b-71f91dfd24e6-metrics-server-audit-profiles\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532449 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84lw9\" (UniqueName: \"kubernetes.io/projected/ad58434d-62ed-4820-bd0b-71f91dfd24e6-kube-api-access-84lw9\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532449 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-secret-metrics-server-tls\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532655 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532472    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-secret-metrics-server-client-certs\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532655 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad58434d-62ed-4820-bd0b-71f91dfd24e6-audit-log\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.532946 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.532856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ad58434d-62ed-4820-bd0b-71f91dfd24e6-audit-log\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.533162 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.533099    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad58434d-62ed-4820-bd0b-71f91dfd24e6-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.534061 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.534020    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ad58434d-62ed-4820-bd0b-71f91dfd24e6-metrics-server-audit-profiles\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.536759 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.536717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-secret-metrics-server-client-certs\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.536847 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.536804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-secret-metrics-server-tls\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.539510 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.539487    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ad58434d-62ed-4820-bd0b-71f91dfd24e6-client-ca-bundle\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.544619 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.542755    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84lw9\" (UniqueName: \"kubernetes.io/projected/ad58434d-62ed-4820-bd0b-71f91dfd24e6-kube-api-access-84lw9\") pod \"metrics-server-ff65875f-tlftv\" (UID: \"ad58434d-62ed-4820-bd0b-71f91dfd24e6\") " pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.628438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.628389    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:47:42.793968 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.793665    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8db66f99c-95smh"]
Jul 28 00:47:42.799860 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.799748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:42.825473 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.825449    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-748b45958f-g8q7r"]
Jul 28 00:47:42.828866 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.828842    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.860669 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.860472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748b45958f-g8q7r"]
Jul 28 00:47:42.866621 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.866565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:42.935562 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-trusted-ca-bundle\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.935845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935578    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-oauth-serving-cert\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.935845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935615    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-serving-cert\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.935845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-oauth-config\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.935845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-console-config\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.935845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9nh9\" (UniqueName: \"kubernetes.io/projected/d823bf33-f89c-451c-93c9-d7c056d260f1-kube-api-access-q9nh9\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.935845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.935757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-service-ca\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:42.949853 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:42.949824    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-ff65875f-tlftv"]
Jul 28 00:47:42.952562 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:42.952518    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad58434d_62ed_4820_bd0b_71f91dfd24e6.slice/crio-433745fcdaa9cfe7fd563ca6db6e9ac80aa465585f4383751e2fdbc761b08a72 WatchSource:0}: Error finding container 433745fcdaa9cfe7fd563ca6db6e9ac80aa465585f4383751e2fdbc761b08a72: Status 404 returned error can't find the container with id 433745fcdaa9cfe7fd563ca6db6e9ac80aa465585f4383751e2fdbc761b08a72
Jul 28 00:47:43.036770 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-serving-cert\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.036884 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-oauth-config\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.036884 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-console-config\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.036884 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9nh9\" (UniqueName: \"kubernetes.io/projected/d823bf33-f89c-451c-93c9-d7c056d260f1-kube-api-access-q9nh9\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.037039 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036908    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-service-ca\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.037039 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036959    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-trusted-ca-bundle\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.037039 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.036985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-oauth-serving-cert\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.037770 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.037742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-oauth-serving-cert\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.038589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.038378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-console-config\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.039441 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.039398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-trusted-ca-bundle\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.039640 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.039617    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-service-ca\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.040553 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.040513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-serving-cert\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.042627 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.042579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-oauth-config\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.045994 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.045953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9nh9\" (UniqueName: \"kubernetes.io/projected/d823bf33-f89c-451c-93c9-d7c056d260f1-kube-api-access-q9nh9\") pod \"console-748b45958f-g8q7r\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") " pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.156245 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.155981    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:43.178205 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.178049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"}
Jul 28 00:47:43.178205 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.178147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"}
Jul 28 00:47:43.178205 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.178187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"}
Jul 28 00:47:43.179604 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.179577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-ff65875f-tlftv" event={"ID":"ad58434d-62ed-4820-bd0b-71f91dfd24e6","Type":"ContainerStarted","Data":"433745fcdaa9cfe7fd563ca6db6e9ac80aa465585f4383751e2fdbc761b08a72"}
Jul 28 00:47:43.299914 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.299882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:43.367033 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.366607    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-748b45958f-g8q7r"]
Jul 28 00:47:43.535366 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.535304    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:43.536420 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.536373    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:43.543000 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.542728    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:43.575395 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.575329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:43.670247 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:43.670162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:43.720077 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:43.719853    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd823bf33_f89c_451c_93c9_d7c056d260f1.slice/crio-ddecd2858d4fc395c0559826b52379fec3c1037151bb24f725b858366bc3c917 WatchSource:0}: Error finding container ddecd2858d4fc395c0559826b52379fec3c1037151bb24f725b858366bc3c917: Status 404 returned error can't find the container with id ddecd2858d4fc395c0559826b52379fec3c1037151bb24f725b858366bc3c917
Jul 28 00:47:44.178150 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.178056    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:44.187396 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.187361    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206201    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"}
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206288    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"}
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f4x8d" event={"ID":"3c50f947-74a7-4285-adaa-0e77f24fc849","Type":"ContainerStarted","Data":"79ddd8bed71b706f8aa3c39d1b0830af6f81b8bdee026a86e9af1862df40f3a4"}
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206321    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748b45958f-g8q7r" event={"ID":"d823bf33-f89c-451c-93c9-d7c056d260f1","Type":"ContainerStarted","Data":"c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd"}
Jul 28 00:47:44.206394 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.206358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748b45958f-g8q7r" event={"ID":"d823bf33-f89c-451c-93c9-d7c056d260f1","Type":"ContainerStarted","Data":"ddecd2858d4fc395c0559826b52379fec3c1037151bb24f725b858366bc3c917"}
Jul 28 00:47:44.210269 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.209848    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 00:47:44.210269 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.209996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 00:47:44.210269 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.210150    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 00:47:44.210653 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.210577    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:47:44.214158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.213387    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 00:47:44.214158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.213460    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 00:47:44.214158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.213642    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 00:47:44.214158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.213687    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 00:47:44.214158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.213949    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.214545    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.214659    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.214785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-953br5bvlg17p\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.214884    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.215016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.215116    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xzvnr\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.215531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 00:47:44.216537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.216345    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 00:47:44.229684 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.229591    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-748b45958f-g8q7r" podStartSLOduration=2.22957701 podStartE2EDuration="2.22957701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:44.229463591 +0000 UTC m=+68.978602318" watchObservedRunningTime="2026-07-28 00:47:44.22957701 +0000 UTC m=+68.978715738"
Jul 28 00:47:44.272322 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.272261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:44.350938 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.350902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.350957    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351594 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351565    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351779 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351779 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351894 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351795    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351894 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351820    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.351894 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351833    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drf7d\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-kube-api-access-drf7d\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352038 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.351897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.352079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.352120    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.352154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352411 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.352246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352411 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.352274    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.352411 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.352337    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.372074 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.372039    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:44.453260 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453260 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453354    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453435 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453425    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453443    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453479    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drf7d\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-kube-api-access-drf7d\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453745    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.453863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.453775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.454399 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.454017    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.454399 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.454340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.455053 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.455029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.455984 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.455950    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.456877 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.456837    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.457818 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.457791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.457999 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.457979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.458714 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.458688    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.459629 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.459608    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.460094 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.460072    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.460521 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.460494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.461065 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.461040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.461360 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.461335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.461802 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.461784    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config-out\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.462431 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.462406    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.462913 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.462890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.462983 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.462915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.463804 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.463743    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-web-config\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.464469 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.464449    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.468329 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.468300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drf7d\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-kube-api-access-drf7d\") pod \"prometheus-k8s-0\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.523477 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.523448    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:44.776536 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.776442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:44.877422 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:44.877389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:45.206598 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.206534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:45.313997 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.313965    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:45.423366 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.423329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:45.475446 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.475365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:45.573823 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.573785    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:47:45.787256 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.787171    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:45.836601 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:45.836568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:46.787521 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:46.787486    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:46.888943 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:46.888901    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:46.987703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:46.987669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:47.122803 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:47.122770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:47.226996 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:47.226957    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:47.722989 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:47.722967    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:47:47.724957 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:47:47.724919    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc2bec27_d8b2_46ee_aadf_8f7c174c1a32.slice/crio-fd7d8c5c5e8a229ca1aab9919e246fac1e5c2c18e085b0eb6ff7e9c1db6acc6b WatchSource:0}: Error finding container fd7d8c5c5e8a229ca1aab9919e246fac1e5c2c18e085b0eb6ff7e9c1db6acc6b: Status 404 returned error can't find the container with id fd7d8c5c5e8a229ca1aab9919e246fac1e5c2c18e085b0eb6ff7e9c1db6acc6b
Jul 28 00:47:48.212899 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.212819    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zw59q" event={"ID":"5cd31cd7-5079-487f-9a62-06a8854a3f58","Type":"ContainerStarted","Data":"99a6dcff9d7bb8fdf841feeab8d0e2a0d5c66e6ec892857eaed14369aebdc226"}
Jul 28 00:47:48.216126 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.216102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerStarted","Data":"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"}
Jul 28 00:47:48.217699 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.217673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-l6shq" event={"ID":"8e2d0561-a486-4f89-a50e-9f9e5cbfe537","Type":"ContainerStarted","Data":"d81d785e98fade6a09968f85ba062cc77799c3de37fc8851823681111ce9ab41"}
Jul 28 00:47:48.217819 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.217799    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:47:48.219452 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.219429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-f4x8d" event={"ID":"3c50f947-74a7-4285-adaa-0e77f24fc849","Type":"ContainerStarted","Data":"a720b1931fbdbb01ae0f273006b0638066ed7dbd84a6f67945d0f626f4f61d59"}
Jul 28 00:47:48.220863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.220840    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-ff65875f-tlftv" event={"ID":"ad58434d-62ed-4820-bd0b-71f91dfd24e6","Type":"ContainerStarted","Data":"e53d6b2dd281dd4cfa656bf9ee93251f28d052273a84704f1652d66e2d1f56c4"}
Jul 28 00:47:48.222429 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.222404    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484" exitCode=0
Jul 28 00:47:48.222511 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.222453    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484"}
Jul 28 00:47:48.222598 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.222510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"fd7d8c5c5e8a229ca1aab9919e246fac1e5c2c18e085b0eb6ff7e9c1db6acc6b"}
Jul 28 00:47:48.227950 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.227890    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zw59q" podStartSLOduration=66.397269981 podStartE2EDuration="1m12.227877775s" totalImagesPullingTime="5.830607794s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:48.226967356 +0000 UTC m=+72.976106081" watchObservedRunningTime="2026-07-28 00:47:48.227877775 +0000 UTC m=+72.977016499"
Jul 28 00:47:48.242232 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.242190    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-l6shq" podStartSLOduration=66.487528359 podStartE2EDuration="1m12.242175158s" totalImagesPullingTime="5.754646799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:48.241453987 +0000 UTC m=+72.990592716" watchObservedRunningTime="2026-07-28 00:47:48.242175158 +0000 UTC m=+72.991313881"
Jul 28 00:47:48.257768 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.257735    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-ff65875f-tlftv" podStartSLOduration=1.646288529 podStartE2EDuration="6.25772296s" totalImagesPullingTime="4.611434431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:48.256511644 +0000 UTC m=+73.005650369" watchObservedRunningTime="2026-07-28 00:47:48.25772296 +0000 UTC m=+73.006861739"
Jul 28 00:47:48.299919 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.299876    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-f4x8d" podStartSLOduration=70.299905006 podStartE2EDuration="1m12.299860303s" totalImagesPullingTime="1.999955297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:46:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:48.299722652 +0000 UTC m=+73.048861383" watchObservedRunningTime="2026-07-28 00:47:48.299860303 +0000 UTC m=+73.048999022"
Jul 28 00:47:48.325814 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.325779    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.229177026 podStartE2EDuration="9.325766353s" totalImagesPullingTime="7.078481427s" totalInitContainerRuntime="18.1079ms" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:48.325068799 +0000 UTC m=+73.074207525" watchObservedRunningTime="2026-07-28 00:47:48.325766353 +0000 UTC m=+73.074905106"
Jul 28 00:47:48.446153 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.446120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:48.546939 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:48.546867    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:51.237796 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:51.237762    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb"}
Jul 28 00:47:51.238149 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:51.237854    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b"}
Jul 28 00:47:52.257093 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:52.257061    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:53.157308 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.157250    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:53.157400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.157314    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:53.162250 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.162214    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:53.249805 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.249776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e"}
Jul 28 00:47:53.249909 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.249833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635"}
Jul 28 00:47:53.249909 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.249862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719"}
Jul 28 00:47:53.249909 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.249887    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerStarted","Data":"c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747"}
Jul 28 00:47:53.253570 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.253551    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:47:53.277621 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.277420    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.742141593 podStartE2EDuration="9.277395569s" totalImagesPullingTime="4.514841638s" totalInitContainerRuntime="20.412338ms" isStatefulPod=false podCreationTimestamp="2026-07-28 00:47:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:47:53.275537841 +0000 UTC m=+78.024676564" watchObservedRunningTime="2026-07-28 00:47:53.277395569 +0000 UTC m=+78.026534295"
Jul 28 00:47:53.328260 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.328212    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b74b69b8f-rpk8b"]
Jul 28 00:47:53.387081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.387055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:53.489100 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.489038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:53.593191 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.593166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:53.688784 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:53.688761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:54.282887 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:54.282858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:54.383386 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:54.383358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:54.523766 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:54.523720    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:47:54.797319 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:54.797269    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:47:54.855772 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:47:54.855747    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:02.628870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:02.628755    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:48:02.628870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:02.628813    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:48:07.822782 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:07.822746    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8db66f99c-95smh" podUID="82d3863f-96e7-4145-a4b6-d4c474e48261" containerName="console" containerID="cri-o://af750b9970e34ad4d2743e45e57e39495b0edacd77b462e2acd448fa01569105" gracePeriod=15
Jul 28 00:48:08.307658 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.307629    2576 generic.go:356] "Generic (PLEG): container finished" podID="82d3863f-96e7-4145-a4b6-d4c474e48261" containerID="af750b9970e34ad4d2743e45e57e39495b0edacd77b462e2acd448fa01569105" exitCode=0
Jul 28 00:48:08.307805 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.307707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8db66f99c-95smh" event={"ID":"82d3863f-96e7-4145-a4b6-d4c474e48261","Type":"ContainerDied","Data":"af750b9970e34ad4d2743e45e57e39495b0edacd77b462e2acd448fa01569105"}
Jul 28 00:48:08.623958 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.623931    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:48:08.660081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660055    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-oauth-config\") pod \"82d3863f-96e7-4145-a4b6-d4c474e48261\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") "
Jul 28 00:48:08.660251 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660089    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-service-ca\") pod \"82d3863f-96e7-4145-a4b6-d4c474e48261\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") "
Jul 28 00:48:08.660251 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660112    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-console-config\") pod \"82d3863f-96e7-4145-a4b6-d4c474e48261\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") "
Jul 28 00:48:08.660251 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-serving-cert\") pod \"82d3863f-96e7-4145-a4b6-d4c474e48261\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") "
Jul 28 00:48:08.660251 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660205    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88f7n\" (UniqueName: \"kubernetes.io/projected/82d3863f-96e7-4145-a4b6-d4c474e48261-kube-api-access-88f7n\") pod \"82d3863f-96e7-4145-a4b6-d4c474e48261\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") "
Jul 28 00:48:08.660498 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660255    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-oauth-serving-cert\") pod \"82d3863f-96e7-4145-a4b6-d4c474e48261\" (UID: \"82d3863f-96e7-4145-a4b6-d4c474e48261\") "
Jul 28 00:48:08.660498 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660463    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-console-config" (OuterVolumeSpecName: "console-config") pod "82d3863f-96e7-4145-a4b6-d4c474e48261" (UID: "82d3863f-96e7-4145-a4b6-d4c474e48261"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:08.660498 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660483    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-service-ca" (OuterVolumeSpecName: "service-ca") pod "82d3863f-96e7-4145-a4b6-d4c474e48261" (UID: "82d3863f-96e7-4145-a4b6-d4c474e48261"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:08.660859 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.660816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82d3863f-96e7-4145-a4b6-d4c474e48261" (UID: "82d3863f-96e7-4145-a4b6-d4c474e48261"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:08.662488 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.662448    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82d3863f-96e7-4145-a4b6-d4c474e48261" (UID: "82d3863f-96e7-4145-a4b6-d4c474e48261"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:08.662488 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.662467    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d3863f-96e7-4145-a4b6-d4c474e48261-kube-api-access-88f7n" (OuterVolumeSpecName: "kube-api-access-88f7n") pod "82d3863f-96e7-4145-a4b6-d4c474e48261" (UID: "82d3863f-96e7-4145-a4b6-d4c474e48261"). InnerVolumeSpecName "kube-api-access-88f7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:48:08.662610 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.662495    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82d3863f-96e7-4145-a4b6-d4c474e48261" (UID: "82d3863f-96e7-4145-a4b6-d4c474e48261"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:08.761171 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.761141    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:08.761171 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.761167    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:08.761349 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.761182    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:08.761349 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.761194    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82d3863f-96e7-4145-a4b6-d4c474e48261-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:08.761349 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.761205    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82d3863f-96e7-4145-a4b6-d4c474e48261-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:08.761349 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:08.761217    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88f7n\" (UniqueName: \"kubernetes.io/projected/82d3863f-96e7-4145-a4b6-d4c474e48261-kube-api-access-88f7n\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:09.290381 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.290351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:09.312738 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.312713    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8db66f99c-95smh"
Jul 28 00:48:09.312861 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.312715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8db66f99c-95smh" event={"ID":"82d3863f-96e7-4145-a4b6-d4c474e48261","Type":"ContainerDied","Data":"72171cdaca3918c04004091de9a050b22b9ccae16dd2ee01d678f34a988e35e6"}
Jul 28 00:48:09.312904 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.312865    2576 scope.go:117] "RemoveContainer" containerID="af750b9970e34ad4d2743e45e57e39495b0edacd77b462e2acd448fa01569105"
Jul 28 00:48:09.341343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.341294    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8db66f99c-95smh"]
Jul 28 00:48:09.342526 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.342502    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8db66f99c-95smh"]
Jul 28 00:48:09.342825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.342808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:09.749204 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:09.749163    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d3863f-96e7-4145-a4b6-d4c474e48261" path="/var/lib/kubelet/pods/82d3863f-96e7-4145-a4b6-d4c474e48261/volumes"
Jul 28 00:48:18.349944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.349907    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b74b69b8f-rpk8b" podUID="b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" containerName="console" containerID="cri-o://cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb" gracePeriod=15
Jul 28 00:48:18.612941 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.612885    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:48:18.646265 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646239    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-oauth-config\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646398 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646304    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvvhw\" (UniqueName: \"kubernetes.io/projected/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-kube-api-access-vvvhw\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646398 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-serving-cert\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646495 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646429    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-trusted-ca-bundle\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646534 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646498    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-config\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646578 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646535    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-oauth-serving-cert\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646578 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-service-ca\") pod \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\" (UID: \"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4\") "
Jul 28 00:48:18.646988 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.646952    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:18.647120 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.647085    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:18.647120 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.647092    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-service-ca" (OuterVolumeSpecName: "service-ca") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:18.647333 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.647179    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-config" (OuterVolumeSpecName: "console-config") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:18.648561 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.648529    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:18.648663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.648602    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-kube-api-access-vvvhw" (OuterVolumeSpecName: "kube-api-access-vvvhw") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "kube-api-access-vvvhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:48:18.648843 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.648824    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" (UID: "b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:18.747601 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747579    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:18.747601 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747599    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:18.747715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747608    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:18.747715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747617    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:18.747715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747625    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvvhw\" (UniqueName: \"kubernetes.io/projected/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-kube-api-access-vvvhw\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:18.747715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747633    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:18.747715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:18.747641    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:19.228973 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.228941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-l6shq"
Jul 28 00:48:19.350873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.350837    2576 generic.go:356] "Generic (PLEG): container finished" podID="b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" containerID="cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb" exitCode=0
Jul 28 00:48:19.351274 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.350883    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b74b69b8f-rpk8b"
Jul 28 00:48:19.351274 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.350922    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74b69b8f-rpk8b" event={"ID":"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4","Type":"ContainerDied","Data":"cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb"}
Jul 28 00:48:19.351274 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.350968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b74b69b8f-rpk8b" event={"ID":"b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4","Type":"ContainerDied","Data":"2d3ce682dfc2e3ee89f8846116bed36366425a2e2648dc49ecff2696b710dc9f"}
Jul 28 00:48:19.351274 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.350988    2576 scope.go:117] "RemoveContainer" containerID="cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb"
Jul 28 00:48:19.359754 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.359738    2576 scope.go:117] "RemoveContainer" containerID="cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb"
Jul 28 00:48:19.360019 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:19.360000    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb\": container with ID starting with cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb not found: ID does not exist" containerID="cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb"
Jul 28 00:48:19.360062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.360026    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb"} err="failed to get container status \"cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb\": rpc error: code = NotFound desc = could not find container \"cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb\": container with ID starting with cfffd291bc7d2886f024bfb9b9fe6b25a6163d6b073c1bf2d1a617a6cb81eacb not found: ID does not exist"
Jul 28 00:48:19.370768 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.370743    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b74b69b8f-rpk8b"]
Jul 28 00:48:19.376712 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.376694    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b74b69b8f-rpk8b"]
Jul 28 00:48:19.749793 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:19.749764    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" path="/var/lib/kubelet/pods/b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4/volumes"
Jul 28 00:48:22.636086 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:22.636052    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:48:22.639964 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:22.639941    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-ff65875f-tlftv"
Jul 28 00:48:24.555159 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:24.555126    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:24.656565 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:24.656536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:29.718907 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:29.718869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:29.771858 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:29.771829    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:44.524551 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:44.524520    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:48:44.543054 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:44.543032    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:48:45.471358 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:45.471332    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:48:57.209660 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:57.209598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:57.511407 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:57.511307    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:48:58.265314 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265280    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 00:48:58.265686 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265586    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="alertmanager" containerID="cri-o://da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350" gracePeriod=120
Jul 28 00:48:58.265762 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265652    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-metric" containerID="cri-o://fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0" gracePeriod=120
Jul 28 00:48:58.265762 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265683    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="prom-label-proxy" containerID="cri-o://bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf" gracePeriod=120
Jul 28 00:48:58.265762 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265701    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-web" containerID="cri-o://2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb" gracePeriod=120
Jul 28 00:48:58.265762 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265718    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="config-reloader" containerID="cri-o://4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8" gracePeriod=120
Jul 28 00:48:58.265921 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.265744    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy" containerID="cri-o://d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483" gracePeriod=120
Jul 28 00:48:58.507330 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507295    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf" exitCode=0
Jul 28 00:48:58.507330 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507320    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483" exitCode=0
Jul 28 00:48:58.507330 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507328    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8" exitCode=0
Jul 28 00:48:58.507532 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507338    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350" exitCode=0
Jul 28 00:48:58.507532 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507374    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"}
Jul 28 00:48:58.507532 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507440    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"}
Jul 28 00:48:58.507532 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507466    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"}
Jul 28 00:48:58.507532 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:58.507491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"}
Jul 28 00:48:59.508854 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.508828    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:48:59.513248 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513209    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0" exitCode=0
Jul 28 00:48:59.513248 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513248    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerID="2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb" exitCode=0
Jul 28 00:48:59.513404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"}
Jul 28 00:48:59.513404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513305    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 00:48:59.513404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"}
Jul 28 00:48:59.513404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513385    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6990f9d-1c41-48b7-8c8f-e4db42135f02","Type":"ContainerDied","Data":"3d18c7791fd3ff874c1d9627d7ebb2365f9903433740a0ddd2bee60dbb556954"}
Jul 28 00:48:59.513404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.513398    2576 scope.go:117] "RemoveContainer" containerID="bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"
Jul 28 00:48:59.521986 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.521973    2576 scope.go:117] "RemoveContainer" containerID="fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"
Jul 28 00:48:59.529341 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.529317    2576 scope.go:117] "RemoveContainer" containerID="d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"
Jul 28 00:48:59.538209 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.538182    2576 scope.go:117] "RemoveContainer" containerID="2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"
Jul 28 00:48:59.545979 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.545953    2576 scope.go:117] "RemoveContainer" containerID="4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"
Jul 28 00:48:59.552805 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.552782    2576 scope.go:117] "RemoveContainer" containerID="da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"
Jul 28 00:48:59.559991 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.559972    2576 scope.go:117] "RemoveContainer" containerID="d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"
Jul 28 00:48:59.566728 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.566711    2576 scope.go:117] "RemoveContainer" containerID="bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"
Jul 28 00:48:59.566970 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.566949    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf\": container with ID starting with bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf not found: ID does not exist" containerID="bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"
Jul 28 00:48:59.567051 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.566979    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"} err="failed to get container status \"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf\": rpc error: code = NotFound desc = could not find container \"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf\": container with ID starting with bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf not found: ID does not exist"
Jul 28 00:48:59.567051 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.566997    2576 scope.go:117] "RemoveContainer" containerID="fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"
Jul 28 00:48:59.567271 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.567252    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0\": container with ID starting with fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0 not found: ID does not exist" containerID="fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"
Jul 28 00:48:59.567343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.567277    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"} err="failed to get container status \"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0\": rpc error: code = NotFound desc = could not find container \"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0\": container with ID starting with fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0 not found: ID does not exist"
Jul 28 00:48:59.567343 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.567293    2576 scope.go:117] "RemoveContainer" containerID="d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"
Jul 28 00:48:59.567528 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.567512    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483\": container with ID starting with d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483 not found: ID does not exist" containerID="d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"
Jul 28 00:48:59.567583 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.567530    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"} err="failed to get container status \"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483\": rpc error: code = NotFound desc = could not find container \"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483\": container with ID starting with d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483 not found: ID does not exist"
Jul 28 00:48:59.567583 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.567544    2576 scope.go:117] "RemoveContainer" containerID="2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"
Jul 28 00:48:59.567784 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.567767    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb\": container with ID starting with 2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb not found: ID does not exist" containerID="2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"
Jul 28 00:48:59.567840 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.567789    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"} err="failed to get container status \"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb\": rpc error: code = NotFound desc = could not find container \"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb\": container with ID starting with 2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb not found: ID does not exist"
Jul 28 00:48:59.567840 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.567804    2576 scope.go:117] "RemoveContainer" containerID="4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"
Jul 28 00:48:59.568015 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.568000    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8\": container with ID starting with 4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8 not found: ID does not exist" containerID="4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"
Jul 28 00:48:59.568075 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568017    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"} err="failed to get container status \"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8\": rpc error: code = NotFound desc = could not find container \"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8\": container with ID starting with 4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8 not found: ID does not exist"
Jul 28 00:48:59.568075 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568028    2576 scope.go:117] "RemoveContainer" containerID="da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"
Jul 28 00:48:59.568269 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.568252    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350\": container with ID starting with da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350 not found: ID does not exist" containerID="da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"
Jul 28 00:48:59.568341 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568271    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"} err="failed to get container status \"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350\": rpc error: code = NotFound desc = could not find container \"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350\": container with ID starting with da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350 not found: ID does not exist"
Jul 28 00:48:59.568341 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568283    2576 scope.go:117] "RemoveContainer" containerID="d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"
Jul 28 00:48:59.568524 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:48:59.568508    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99\": container with ID starting with d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99 not found: ID does not exist" containerID="d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"
Jul 28 00:48:59.568588 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568526    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"} err="failed to get container status \"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99\": rpc error: code = NotFound desc = could not find container \"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99\": container with ID starting with d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99 not found: ID does not exist"
Jul 28 00:48:59.568588 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568538    2576 scope.go:117] "RemoveContainer" containerID="bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"
Jul 28 00:48:59.568745 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568725    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf"} err="failed to get container status \"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf\": rpc error: code = NotFound desc = could not find container \"bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf\": container with ID starting with bb0eacc306ff1dd8f1f918b48871331786d7f3914217fd3a937ffb90addacfbf not found: ID does not exist"
Jul 28 00:48:59.568798 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568744    2576 scope.go:117] "RemoveContainer" containerID="fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"
Jul 28 00:48:59.568967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568940    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0"} err="failed to get container status \"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0\": rpc error: code = NotFound desc = could not find container \"fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0\": container with ID starting with fe987098429fb22d87024e6c1d41b9a0f5ddb12eac890a31913786348565c8b0 not found: ID does not exist"
Jul 28 00:48:59.568967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.568959    2576 scope.go:117] "RemoveContainer" containerID="d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"
Jul 28 00:48:59.569186 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569154    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483"} err="failed to get container status \"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483\": rpc error: code = NotFound desc = could not find container \"d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483\": container with ID starting with d39bc0670bb9baa0c422c5379fc114272a6a7dad8f6670af4399156aac762483 not found: ID does not exist"
Jul 28 00:48:59.569280 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569186    2576 scope.go:117] "RemoveContainer" containerID="2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"
Jul 28 00:48:59.569410 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569392    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb"} err="failed to get container status \"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb\": rpc error: code = NotFound desc = could not find container \"2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb\": container with ID starting with 2c171764b3eaba7bead8fd9e4355a0aa0fd092d6ecdbe5c89d9dc516c41c27cb not found: ID does not exist"
Jul 28 00:48:59.569473 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569408    2576 scope.go:117] "RemoveContainer" containerID="4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"
Jul 28 00:48:59.569625 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569606    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8"} err="failed to get container status \"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8\": rpc error: code = NotFound desc = could not find container \"4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8\": container with ID starting with 4bed1c542a1bfb94e56a6f553cc7a1816eccb93d1baa938deaaf4112cd4b37f8 not found: ID does not exist"
Jul 28 00:48:59.569681 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569625    2576 scope.go:117] "RemoveContainer" containerID="da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"
Jul 28 00:48:59.569797 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569777    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350"} err="failed to get container status \"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350\": rpc error: code = NotFound desc = could not find container \"da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350\": container with ID starting with da94fd966b2b2a37bf878d7c1e2d9b826d8fe2a83b092001abad53f46c48a350 not found: ID does not exist"
Jul 28 00:48:59.569853 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.569797    2576 scope.go:117] "RemoveContainer" containerID="d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"
Jul 28 00:48:59.570028 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.570010    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99"} err="failed to get container status \"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99\": rpc error: code = NotFound desc = could not find container \"d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99\": container with ID starting with d6c5f9e783c5048d640b1c165b1af70dcfcdb41c2e03eaf49b4868b578260c99 not found: ID does not exist"
Jul 28 00:48:59.572879 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.572860    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-trusted-ca-bundle\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.572963 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.572895    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj9w5\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-kube-api-access-hj9w5\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.572963 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.572922    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-main-tls\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.572963 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.572941    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.572975    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-metrics-client-ca\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573002    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-out\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573024    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-web-config\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573079    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-volume\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573106    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-cluster-tls-config\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573132    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-web\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573169    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573198    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-tls-assets\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573255    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-main-db\") pod \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\" (UID: \"c6990f9d-1c41-48b7-8c8f-e4db42135f02\") "
Jul 28 00:48:59.573854 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573813    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:48:59.574067 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.573688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:59.574145 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.574065    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:48:59.576122 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.576069    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.576302 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.576276    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.576579 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.576486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:48:59.576579 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.576538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-volume" (OuterVolumeSpecName: "config-volume") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.576579 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.576562    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-out" (OuterVolumeSpecName: "config-out") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:48:59.576765 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.576703    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.577046 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.577027    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-kube-api-access-hj9w5" (OuterVolumeSpecName: "kube-api-access-hj9w5") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "kube-api-access-hj9w5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:48:59.577480 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.577460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.583186 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.583161    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.588878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.588827    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-web-config" (OuterVolumeSpecName: "web-config") pod "c6990f9d-1c41-48b7-8c8f-e4db42135f02" (UID: "c6990f9d-1c41-48b7-8c8f-e4db42135f02"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:48:59.674298 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674272    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674298 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674294    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-tls-assets\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674304    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-main-db\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674314    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674322    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj9w5\" (UniqueName: \"kubernetes.io/projected/c6990f9d-1c41-48b7-8c8f-e4db42135f02-kube-api-access-hj9w5\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674331    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-main-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674339    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674349    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6990f9d-1c41-48b7-8c8f-e4db42135f02-metrics-client-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674357    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-out\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674365    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-web-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674372    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674379    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-cluster-tls-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.674419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.674387    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6990f9d-1c41-48b7-8c8f-e4db42135f02-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:48:59.831951 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.831918    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 00:48:59.835357 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:48:59.835331    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 00:49:01.749300 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:01.749266    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" path="/var/lib/kubelet/pods/c6990f9d-1c41-48b7-8c8f-e4db42135f02/volumes"
Jul 28 00:49:02.617082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617048    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:49:02.617392 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617351    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="prometheus" containerID="cri-o://0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b" gracePeriod=600
Jul 28 00:49:02.617537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617395    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="thanos-sidecar" containerID="cri-o://c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747" gracePeriod=600
Jul 28 00:49:02.617537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617394    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy" containerID="cri-o://aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635" gracePeriod=600
Jul 28 00:49:02.617537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617439    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-web" containerID="cri-o://e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719" gracePeriod=600
Jul 28 00:49:02.617537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617442    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="config-reloader" containerID="cri-o://8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb" gracePeriod=600
Jul 28 00:49:02.617537 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:02.617436    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-thanos" containerID="cri-o://c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e" gracePeriod=600
Jul 28 00:49:03.530873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530843    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e" exitCode=0
Jul 28 00:49:03.530873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530867    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635" exitCode=0
Jul 28 00:49:03.530873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530873    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747" exitCode=0
Jul 28 00:49:03.530873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530879    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb" exitCode=0
Jul 28 00:49:03.530873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530884    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b" exitCode=0
Jul 28 00:49:03.531368 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530907    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e"}
Jul 28 00:49:03.531368 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635"}
Jul 28 00:49:03.531368 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.530998    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747"}
Jul 28 00:49:03.531368 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.531021    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb"}
Jul 28 00:49:03.531368 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.531046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b"}
Jul 28 00:49:03.953725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:03.953703    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.006383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006353    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-serving-certs-ca-bundle\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006567 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006405    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-tls-assets\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006567 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006497    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-metrics-client-ca\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006567 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006538    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-trusted-ca-bundle\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006567 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006565    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-0\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006597    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006624    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drf7d\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-kube-api-access-drf7d\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006649    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-tls\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006683    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006713    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-db\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006744    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-1\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.006830 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006782    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.007174 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.006839    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.007708 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.007440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.007845    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-thanos-prometheus-http-client-file\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.007896    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-kubelet-serving-ca-bundle\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.007923    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.007933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-kube-rbac-proxy\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.007959    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-2\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008001    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-metrics-client-certs\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008035    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-grpc-tls\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008060    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-web-config\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008103    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config-out\") pod \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\" (UID: \"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32\") "
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008338    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008358    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008379    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-metrics-client-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008394    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008936    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.008945    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.009294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:04.012851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.009630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.010382    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config" (OuterVolumeSpecName: "config") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.010524    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.012456    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.012594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.012701    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config-out" (OuterVolumeSpecName: "config-out") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.012737    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.012952    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.013887    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.013944 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.013909    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-kube-api-access-drf7d" (OuterVolumeSpecName: "kube-api-access-drf7d") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "kube-api-access-drf7d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:49:04.014487 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.013959    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:49:04.014781 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.014755    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.023622 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.023598    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-web-config" (OuterVolumeSpecName: "web-config") pod "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" (UID: "cc2bec27-d8b2-46ee-aadf-8f7c174c1a32"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108857    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-tls-assets\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108880    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108891    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108902    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drf7d\" (UniqueName: \"kubernetes.io/projected/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-kube-api-access-drf7d\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108911    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108919    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.108926 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108927    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-db\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108936    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108944    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108953    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108962    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108970    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-kube-rbac-proxy\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108978    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108986    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-metrics-client-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.108994    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-secret-grpc-tls\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.109021    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-web-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.109212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.109029    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32-config-out\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:04.537280 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.537180    2576 generic.go:356] "Generic (PLEG): container finished" podID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerID="e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719" exitCode=0
Jul 28 00:49:04.537280 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.537261    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719"}
Jul 28 00:49:04.537742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.537298    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.537742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.537332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cc2bec27-d8b2-46ee-aadf-8f7c174c1a32","Type":"ContainerDied","Data":"fd7d8c5c5e8a229ca1aab9919e246fac1e5c2c18e085b0eb6ff7e9c1db6acc6b"}
Jul 28 00:49:04.537742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.537366    2576 scope.go:117] "RemoveContainer" containerID="c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e"
Jul 28 00:49:04.544906 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.544828    2576 scope.go:117] "RemoveContainer" containerID="aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635"
Jul 28 00:49:04.551379 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.551363    2576 scope.go:117] "RemoveContainer" containerID="e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719"
Jul 28 00:49:04.557457 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.557441    2576 scope.go:117] "RemoveContainer" containerID="c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747"
Jul 28 00:49:04.560620 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.560602    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:49:04.564725 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.564708    2576 scope.go:117] "RemoveContainer" containerID="8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb"
Jul 28 00:49:04.565346 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.565329    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:49:04.570856 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.570840    2576 scope.go:117] "RemoveContainer" containerID="0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b"
Jul 28 00:49:04.577379 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.577365    2576 scope.go:117] "RemoveContainer" containerID="ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484"
Jul 28 00:49:04.583476 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.583462    2576 scope.go:117] "RemoveContainer" containerID="c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e"
Jul 28 00:49:04.583721 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.583704    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e\": container with ID starting with c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e not found: ID does not exist" containerID="c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e"
Jul 28 00:49:04.583760 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.583728    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e"} err="failed to get container status \"c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e\": rpc error: code = NotFound desc = could not find container \"c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e\": container with ID starting with c75119f83e4adfee7639a4d8f3f400da890cde9e50d461a7923ff93044b13a7e not found: ID does not exist"
Jul 28 00:49:04.583760 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.583745    2576 scope.go:117] "RemoveContainer" containerID="aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635"
Jul 28 00:49:04.583968 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.583950    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635\": container with ID starting with aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635 not found: ID does not exist" containerID="aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635"
Jul 28 00:49:04.584024 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.583977    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635"} err="failed to get container status \"aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635\": rpc error: code = NotFound desc = could not find container \"aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635\": container with ID starting with aa0dd6636e89d86719407b247fcc36c3252b92328b9b7ec079b499f834f92635 not found: ID does not exist"
Jul 28 00:49:04.584024 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.583998    2576 scope.go:117] "RemoveContainer" containerID="e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719"
Jul 28 00:49:04.584268 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.584213    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719\": container with ID starting with e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719 not found: ID does not exist" containerID="e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719"
Jul 28 00:49:04.584268 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.584246    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719"} err="failed to get container status \"e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719\": rpc error: code = NotFound desc = could not find container \"e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719\": container with ID starting with e566328f1c83e36fc2eabe64981486647aea04703070f1cd1c0591344fa4b719 not found: ID does not exist"
Jul 28 00:49:04.584268 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.584258    2576 scope.go:117] "RemoveContainer" containerID="c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747"
Jul 28 00:49:04.584491 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.584472    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747\": container with ID starting with c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747 not found: ID does not exist" containerID="c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747"
Jul 28 00:49:04.584531 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.584497    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747"} err="failed to get container status \"c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747\": rpc error: code = NotFound desc = could not find container \"c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747\": container with ID starting with c893f6d7034076c3a92b19f07df93ae706edaffbd0fb862a9fcda0556b781747 not found: ID does not exist"
Jul 28 00:49:04.584531 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.584513    2576 scope.go:117] "RemoveContainer" containerID="8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb"
Jul 28 00:49:04.584770 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.584753    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb\": container with ID starting with 8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb not found: ID does not exist" containerID="8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb"
Jul 28 00:49:04.584825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.584775    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb"} err="failed to get container status \"8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb\": rpc error: code = NotFound desc = could not find container \"8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb\": container with ID starting with 8b9f7fbb1105f69779aec2f9f06144e3eefee17e7635c172fc7fba64e494b3fb not found: ID does not exist"
Jul 28 00:49:04.584825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.584791    2576 scope.go:117] "RemoveContainer" containerID="0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b"
Jul 28 00:49:04.585016 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.585001    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b\": container with ID starting with 0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b not found: ID does not exist" containerID="0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b"
Jul 28 00:49:04.585055 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.585019    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b"} err="failed to get container status \"0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b\": rpc error: code = NotFound desc = could not find container \"0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b\": container with ID starting with 0a4671b040264c90a15a8fc60b5e980bd59eca95061dad7cd9da2d2a6f5ef60b not found: ID does not exist"
Jul 28 00:49:04.585055 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.585032    2576 scope.go:117] "RemoveContainer" containerID="ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484"
Jul 28 00:49:04.585264 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:04.585247    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484\": container with ID starting with ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484 not found: ID does not exist" containerID="ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484"
Jul 28 00:49:04.585310 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.585269    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484"} err="failed to get container status \"ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484\": rpc error: code = NotFound desc = could not find container \"ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484\": container with ID starting with ad254183ccd05ee2cfad14acb1d5842bf53bae999c8391a84650d43a25428484 not found: ID does not exist"
Jul 28 00:49:04.588554 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588531    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:49:04.588800 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588785    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="thanos-sidecar"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588804    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="thanos-sidecar"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588820    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d3863f-96e7-4145-a4b6-d4c474e48261" containerName="console"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588829    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d3863f-96e7-4145-a4b6-d4c474e48261" containerName="console"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588839    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="prom-label-proxy"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588847    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="prom-label-proxy"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588857    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="init-config-reloader"
Jul 28 00:49:04.588870 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588865    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="init-config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588874    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-web"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588883    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-web"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588894    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588903    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588914    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-web"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588922    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-web"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588931    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-thanos"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588940    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-thanos"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588957    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588965    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588976    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588984    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.588997    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="init-config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589005    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="init-config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589015    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="alertmanager"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589023    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="alertmanager"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589032    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-metric"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589041    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-metric"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589053    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589062    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589073    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" containerName="console"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589081    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" containerName="console"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589090    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="prometheus"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589098    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="prometheus"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589174    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589185    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9f6a7ae-6f6b-4f5f-86ec-1bc1919f7cc4" containerName="console"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589195    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589202    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="config-reloader"
Jul 28 00:49:04.589199 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589211    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="prometheus"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589240    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-thanos"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589248    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="alertmanager"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589255    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="thanos-sidecar"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589264    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-web"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589273    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d3863f-96e7-4145-a4b6-d4c474e48261" containerName="console"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589282    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="prom-label-proxy"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589290    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy-web"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589297    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" containerName="kube-rbac-proxy"
Jul 28 00:49:04.590517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.589306    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6990f9d-1c41-48b7-8c8f-e4db42135f02" containerName="kube-rbac-proxy-metric"
Jul 28 00:49:04.594686 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.594667    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.597662 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.597327    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 00:49:04.597662 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.597335    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-xzvnr\""
Jul 28 00:49:04.597803 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.597680    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 00:49:04.597803 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.597749    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 00:49:04.598596 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.598574    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 00:49:04.598763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.598741    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 00:49:04.599020 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.599000    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 00:49:04.599122 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.599106    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 00:49:04.599486 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.599458    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 00:49:04.599727 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.598644    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 00:49:04.600190 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.600131    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 00:49:04.601184 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.601162    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 00:49:04.601380 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.601169    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-953br5bvlg17p\""
Jul 28 00:49:04.601480 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.601325    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 00:49:04.602500 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.602481    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 00:49:04.604021 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.604006    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 00:49:04.607206 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.607183    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:49:04.612002 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.611981    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612096 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612018    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612096 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612096 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1247c28d-cab7-4e1e-88ed-39bcde73636a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612257    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-web-config\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1247c28d-cab7-4e1e-88ed-39bcde73636a-config-out\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612678 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612534    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612678 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfnr8\" (UniqueName: \"kubernetes.io/projected/1247c28d-cab7-4e1e-88ed-39bcde73636a-kube-api-access-dfnr8\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612678 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612678 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612645    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-config\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.612678 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.612658    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713241 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713359 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713359 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713286    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713359 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713359 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713336    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1247c28d-cab7-4e1e-88ed-39bcde73636a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-web-config\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1247c28d-cab7-4e1e-88ed-39bcde73636a-config-out\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713564    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713630    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713698    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfnr8\" (UniqueName: \"kubernetes.io/projected/1247c28d-cab7-4e1e-88ed-39bcde73636a-kube-api-access-dfnr8\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-config\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713780    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.713967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713940    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.714423 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.713981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.714423 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.714409    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.715160 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.714809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.715637 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.715612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.716389 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.716367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.716529 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.716499    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.717294 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.717269    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.717538 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.717511    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.717698 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.717644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1247c28d-cab7-4e1e-88ed-39bcde73636a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.717919 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.717898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.718338 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.718292    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-config\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.718443 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.718374    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.718545 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.718523    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1247c28d-cab7-4e1e-88ed-39bcde73636a-config-out\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.718674 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.718655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-web-config\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.718879 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.718864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.719257 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.719241    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/1247c28d-cab7-4e1e-88ed-39bcde73636a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.719493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.719475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/1247c28d-cab7-4e1e-88ed-39bcde73636a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.723057 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.723036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfnr8\" (UniqueName: \"kubernetes.io/projected/1247c28d-cab7-4e1e-88ed-39bcde73636a-kube-api-access-dfnr8\") pod \"prometheus-k8s-0\" (UID: \"1247c28d-cab7-4e1e-88ed-39bcde73636a\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:04.909157 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:04.909130    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:05.054906 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:05.054875    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 00:49:05.058875 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:49:05.058841    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1247c28d_cab7_4e1e_88ed_39bcde73636a.slice/crio-61baefa6e0e36936c34aa22aa39fa901efc7385ce903bfa568335a39d17f727f WatchSource:0}: Error finding container 61baefa6e0e36936c34aa22aa39fa901efc7385ce903bfa568335a39d17f727f: Status 404 returned error can't find the container with id 61baefa6e0e36936c34aa22aa39fa901efc7385ce903bfa568335a39d17f727f
Jul 28 00:49:05.545697 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:05.545661    2576 generic.go:356] "Generic (PLEG): container finished" podID="1247c28d-cab7-4e1e-88ed-39bcde73636a" containerID="55a2e64bee2e57cd4ce8e21a769f9f3fea4003f8757517f1049d28d2d089c0e8" exitCode=0
Jul 28 00:49:05.546082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:05.545735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerDied","Data":"55a2e64bee2e57cd4ce8e21a769f9f3fea4003f8757517f1049d28d2d089c0e8"}
Jul 28 00:49:05.546082 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:05.545802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"61baefa6e0e36936c34aa22aa39fa901efc7385ce903bfa568335a39d17f727f"}
Jul 28 00:49:05.753792 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:05.753501    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc2bec27-d8b2-46ee-aadf-8f7c174c1a32" path="/var/lib/kubelet/pods/cc2bec27-d8b2-46ee-aadf-8f7c174c1a32/volumes"
Jul 28 00:49:06.518173 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.518136    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58888d4f95-xj9nb"]
Jul 28 00:49:06.521258 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.521242    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.531451 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.531427    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58888d4f95-xj9nb"]
Jul 28 00:49:06.553240 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.553189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"b12913ea3ac2b7e368befcd5e46c42c17bd9042a9d8f6019f329f3793c8ac142"}
Jul 28 00:49:06.553587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.553262    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"fbafb048f33c1b7968046424d7f7ec82978b077cae0a19515c876497740dfb74"}
Jul 28 00:49:06.553587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.553315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"3f7ad07f44567e297def99f327ed54ad177bc5e17ca3ef8c28a9d0b2ddb80f92"}
Jul 28 00:49:06.553587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.553344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"5f5d00c6d751e9d28402699127eb7336bd66897d2701da69e15fea7db263196b"}
Jul 28 00:49:06.553587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.553368    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"198b86b0178e130da48b8e6edf07f88604585135d670eee350db0e51e7622836"}
Jul 28 00:49:06.553587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.553393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"1247c28d-cab7-4e1e-88ed-39bcde73636a","Type":"ContainerStarted","Data":"46b8fa7fa7bb1ff50b0b25eaa851fb0be146fb965185ee4fff8fc867106dcfde"}
Jul 28 00:49:06.571493 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.571467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:06.583198 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.583147    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.564634786 podStartE2EDuration="2.58312858s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.493794ms" isStatefulPod=false podCreationTimestamp="2026-07-28 00:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:49:06.581270414 +0000 UTC m=+151.330409142" watchObservedRunningTime="2026-07-28 00:49:06.58312858 +0000 UTC m=+151.332267305"
Jul 28 00:49:06.622066 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.622034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:06.630238 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630196    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-serving-cert\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.630238 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-config\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.630423 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630271    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-trusted-ca-bundle\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.630423 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630370    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-oauth-config\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.630423 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvd5n\" (UniqueName: \"kubernetes.io/projected/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-kube-api-access-qvd5n\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.630584 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630483    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-oauth-serving-cert\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.630742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.630725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-service-ca\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731263 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-oauth-config\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731403 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731268    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvd5n\" (UniqueName: \"kubernetes.io/projected/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-kube-api-access-qvd5n\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731403 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-oauth-serving-cert\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731403 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-service-ca\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731524 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731402    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-serving-cert\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731524 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-config\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.731524 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.731450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-trusted-ca-bundle\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.732092 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.732065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-oauth-serving-cert\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.732198 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.732107    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-service-ca\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.732198 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.732137    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-config\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.732383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.732363    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-trusted-ca-bundle\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.733670 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.733653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-oauth-config\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.733837 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.733819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-serving-cert\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.738780 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.738760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvd5n\" (UniqueName: \"kubernetes.io/projected/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-kube-api-access-qvd5n\") pod \"console-58888d4f95-xj9nb\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") " pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.829491 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.829470    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:06.950392 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:06.950233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58888d4f95-xj9nb"]
Jul 28 00:49:06.952662 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:49:06.952619    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1461d0_3bd7_42fe_aebc_c6f8769285a3.slice/crio-bfae7bf4ae9ee16b9ae5cd960c033ac74f28b45517b649957a0c2746baa1f61e WatchSource:0}: Error finding container bfae7bf4ae9ee16b9ae5cd960c033ac74f28b45517b649957a0c2746baa1f61e: Status 404 returned error can't find the container with id bfae7bf4ae9ee16b9ae5cd960c033ac74f28b45517b649957a0c2746baa1f61e
Jul 28 00:49:07.559453 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:07.559415    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58888d4f95-xj9nb" event={"ID":"fe1461d0-3bd7-42fe-aebc-c6f8769285a3","Type":"ContainerStarted","Data":"3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598"}
Jul 28 00:49:07.559813 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:07.559470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58888d4f95-xj9nb" event={"ID":"fe1461d0-3bd7-42fe-aebc-c6f8769285a3","Type":"ContainerStarted","Data":"bfae7bf4ae9ee16b9ae5cd960c033ac74f28b45517b649957a0c2746baa1f61e"}
Jul 28 00:49:07.578249 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:07.578185    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58888d4f95-xj9nb" podStartSLOduration=1.578170082 podStartE2EDuration="1.578170082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:49:07.577642363 +0000 UTC m=+152.326781084" watchObservedRunningTime="2026-07-28 00:49:07.578170082 +0000 UTC m=+152.327308806"
Jul 28 00:49:09.909906 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:09.909876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:49:16.829983 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:16.829937    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:16.830473 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:16.830020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:16.834816 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:16.834793    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:17.600959 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:17.600930    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:49:17.644510 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:17.644480    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748b45958f-g8q7r"]
Jul 28 00:49:17.741904 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:17.741876    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:17.842772 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:17.842741    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:17.890305 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:17.890184    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:24.280262 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:24.280216    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:42.664033 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:42.663978    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-748b45958f-g8q7r" podUID="d823bf33-f89c-451c-93c9-d7c056d260f1" containerName="console" containerID="cri-o://c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd" gracePeriod=15
Jul 28 00:49:42.908952 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:42.908930    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:49:43.013451 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013388    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-oauth-serving-cert\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013451 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013427    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-service-ca\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013611 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013454    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-oauth-config\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013611 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013559    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9nh9\" (UniqueName: \"kubernetes.io/projected/d823bf33-f89c-451c-93c9-d7c056d260f1-kube-api-access-q9nh9\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013719 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013610    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-console-config\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013719 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013642    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-trusted-ca-bundle\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013719 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013693    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-serving-cert\") pod \"d823bf33-f89c-451c-93c9-d7c056d260f1\" (UID: \"d823bf33-f89c-451c-93c9-d7c056d260f1\") "
Jul 28 00:49:43.013872 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013853    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:43.013955 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013925    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-console-config" (OuterVolumeSpecName: "console-config") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:43.013955 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.013946    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-service-ca" (OuterVolumeSpecName: "service-ca") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:43.014061 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.014005    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.014061 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.014036    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:49:43.015739 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.015705    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:43.015874 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.015854    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:49:43.015935 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.015913    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d823bf33-f89c-451c-93c9-d7c056d260f1-kube-api-access-q9nh9" (OuterVolumeSpecName: "kube-api-access-q9nh9") pod "d823bf33-f89c-451c-93c9-d7c056d260f1" (UID: "d823bf33-f89c-451c-93c9-d7c056d260f1"). InnerVolumeSpecName "kube-api-access-q9nh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:49:43.114809 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.114784    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.114809 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.114810    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.114984 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.114823    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d823bf33-f89c-451c-93c9-d7c056d260f1-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.114984 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.114835    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9nh9\" (UniqueName: \"kubernetes.io/projected/d823bf33-f89c-451c-93c9-d7c056d260f1-kube-api-access-q9nh9\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.114984 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.114848    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.114984 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.114859    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d823bf33-f89c-451c-93c9-d7c056d260f1-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:49:43.692157 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.692118    2576 generic.go:356] "Generic (PLEG): container finished" podID="d823bf33-f89c-451c-93c9-d7c056d260f1" containerID="c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd" exitCode=0
Jul 28 00:49:43.692616 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.692169    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-748b45958f-g8q7r"
Jul 28 00:49:43.692616 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.692186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748b45958f-g8q7r" event={"ID":"d823bf33-f89c-451c-93c9-d7c056d260f1","Type":"ContainerDied","Data":"c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd"}
Jul 28 00:49:43.692616 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.692259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-748b45958f-g8q7r" event={"ID":"d823bf33-f89c-451c-93c9-d7c056d260f1","Type":"ContainerDied","Data":"ddecd2858d4fc395c0559826b52379fec3c1037151bb24f725b858366bc3c917"}
Jul 28 00:49:43.692616 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.692290    2576 scope.go:117] "RemoveContainer" containerID="c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd"
Jul 28 00:49:43.700721 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.700704    2576 scope.go:117] "RemoveContainer" containerID="c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd"
Jul 28 00:49:43.700954 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:49:43.700936    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd\": container with ID starting with c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd not found: ID does not exist" containerID="c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd"
Jul 28 00:49:43.700997 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.700964    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd"} err="failed to get container status \"c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd\": rpc error: code = NotFound desc = could not find container \"c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd\": container with ID starting with c5204d86cba5686d4f3692a878a545d72dec4b063aa09aa65b2e4bb1d48ddcdd not found: ID does not exist"
Jul 28 00:49:43.712116 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.712096    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-748b45958f-g8q7r"]
Jul 28 00:49:43.715375 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.715353    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-748b45958f-g8q7r"]
Jul 28 00:49:43.749154 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:43.749130    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d823bf33-f89c-451c-93c9-d7c056d260f1" path="/var/lib/kubelet/pods/d823bf33-f89c-451c-93c9-d7c056d260f1/volumes"
Jul 28 00:49:59.774123 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:59.774084    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:49:59.874736 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:49:59.874708    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:04.910181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:04.910139    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:50:04.924851 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:04.924824    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:50:05.785239 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:05.785197    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 00:50:12.447803 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:12.447765    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:12.529986 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:12.529954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:16.115419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.115374    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f7f9b4999-zs5gb"]
Jul 28 00:50:16.115899 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.115776    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d823bf33-f89c-451c-93c9-d7c056d260f1" containerName="console"
Jul 28 00:50:16.115899 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.115793    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d823bf33-f89c-451c-93c9-d7c056d260f1" containerName="console"
Jul 28 00:50:16.115899 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.115877    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d823bf33-f89c-451c-93c9-d7c056d260f1" containerName="console"
Jul 28 00:50:16.119042 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.119020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.129724 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.129704    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7f9b4999-zs5gb"]
Jul 28 00:50:16.147897 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.147873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-serving-cert\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.148063 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.147904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-oauth-serving-cert\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.148063 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.147939    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-oauth-config\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.148063 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.148028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-service-ca\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.148245 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.148065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-config\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.148245 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.148095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdnxk\" (UniqueName: \"kubernetes.io/projected/65489689-9fa0-43fe-82f7-b583ff91b1fb-kube-api-access-cdnxk\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.148245 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.148140    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-trusted-ca-bundle\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.170751 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.170727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:16.249197 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-serving-cert\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.249344 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-oauth-serving-cert\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.249344 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-oauth-config\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.249344 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249331    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-service-ca\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.249508 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-config\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.249508 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdnxk\" (UniqueName: \"kubernetes.io/projected/65489689-9fa0-43fe-82f7-b583ff91b1fb-kube-api-access-cdnxk\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.249508 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.249435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-trusted-ca-bundle\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.250112 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.250085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-service-ca\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.250210 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.250091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-oauth-serving-cert\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.250210 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.250091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-config\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.250407 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.250389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-trusted-ca-bundle\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.251547 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.251518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-oauth-config\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.251631 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.251619    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-serving-cert\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.256723 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.256702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdnxk\" (UniqueName: \"kubernetes.io/projected/65489689-9fa0-43fe-82f7-b583ff91b1fb-kube-api-access-cdnxk\") pod \"console-5f7f9b4999-zs5gb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") " pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.270712 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.270693    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:16.371450 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.371392    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:16.428052 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.428031    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:16.548096 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.547983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f7f9b4999-zs5gb"]
Jul 28 00:50:16.550167 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:50:16.550143    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65489689_9fa0_43fe_82f7_b583ff91b1fb.slice/crio-fd53380333ef91b6dce1ba056d7b04bc37e65dd458ffbe129d1cfe0f61640a5c WatchSource:0}: Error finding container fd53380333ef91b6dce1ba056d7b04bc37e65dd458ffbe129d1cfe0f61640a5c: Status 404 returned error can't find the container with id fd53380333ef91b6dce1ba056d7b04bc37e65dd458ffbe129d1cfe0f61640a5c
Jul 28 00:50:16.809009 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.808975    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f9b4999-zs5gb" event={"ID":"65489689-9fa0-43fe-82f7-b583ff91b1fb","Type":"ContainerStarted","Data":"c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24"}
Jul 28 00:50:16.809168 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.809031    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f9b4999-zs5gb" event={"ID":"65489689-9fa0-43fe-82f7-b583ff91b1fb","Type":"ContainerStarted","Data":"fd53380333ef91b6dce1ba056d7b04bc37e65dd458ffbe129d1cfe0f61640a5c"}
Jul 28 00:50:16.826013 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.825945    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5f7f9b4999-zs5gb" podStartSLOduration=0.82593076 podStartE2EDuration="825.93076ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:50:16.825097724 +0000 UTC m=+221.574236462" watchObservedRunningTime="2026-07-28 00:50:16.82593076 +0000 UTC m=+221.575069484"
Jul 28 00:50:16.928482 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:16.928451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:17.032867 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:17.032838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:23.784608 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:23.784575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:24.310510 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:24.310476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:24.412158 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:24.412129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:26.428520 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:26.428480    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:26.428929 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:26.428579    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:26.433107 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:26.433086    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:26.847304 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:26.847272    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:50:26.889701 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:26.889671    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58888d4f95-xj9nb"]
Jul 28 00:50:26.987074 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:26.987047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:27.038272 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:27.038244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:27.087150 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:27.087116    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:39.295521 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:39.295459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:39.395880 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:39.395852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:50:51.908539 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:51.908482    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58888d4f95-xj9nb" podUID="fe1461d0-3bd7-42fe-aebc-c6f8769285a3" containerName="console" containerID="cri-o://3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598" gracePeriod=15
Jul 28 00:50:52.140035 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.140013    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:50:52.210326 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210264    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-oauth-serving-cert\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210326 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210307    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-trusted-ca-bundle\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210489 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210336    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-oauth-config\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210489 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210458    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qvd5n\" (UniqueName: \"kubernetes.io/projected/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-kube-api-access-qvd5n\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210590 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210521    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-config\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210590 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210574    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-serving-cert\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210696 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210606    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-service-ca\") pod \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\" (UID: \"fe1461d0-3bd7-42fe-aebc-c6f8769285a3\") "
Jul 28 00:50:52.210749 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:50:52.210810 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210775    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:50:52.210933 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210917    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.211003 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210922    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-config" (OuterVolumeSpecName: "console-config") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:50:52.211003 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.210943    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.211104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.211038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:50:52.212561 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.212525    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:50:52.212561 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.212536    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:50:52.212692 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.212538    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-kube-api-access-qvd5n" (OuterVolumeSpecName: "kube-api-access-qvd5n") pod "fe1461d0-3bd7-42fe-aebc-c6f8769285a3" (UID: "fe1461d0-3bd7-42fe-aebc-c6f8769285a3"). InnerVolumeSpecName "kube-api-access-qvd5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:50:52.311995 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.311970    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.311995 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.311989    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvd5n\" (UniqueName: \"kubernetes.io/projected/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-kube-api-access-qvd5n\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.312128 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.311999    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.312128 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.312008    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.312128 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.312017    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1461d0-3bd7-42fe-aebc-c6f8769285a3-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:50:52.932753 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.932721    2576 generic.go:356] "Generic (PLEG): container finished" podID="fe1461d0-3bd7-42fe-aebc-c6f8769285a3" containerID="3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598" exitCode=0
Jul 28 00:50:52.933136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.932762    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58888d4f95-xj9nb"
Jul 28 00:50:52.933136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.932774    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58888d4f95-xj9nb" event={"ID":"fe1461d0-3bd7-42fe-aebc-c6f8769285a3","Type":"ContainerDied","Data":"3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598"}
Jul 28 00:50:52.933136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.932811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58888d4f95-xj9nb" event={"ID":"fe1461d0-3bd7-42fe-aebc-c6f8769285a3","Type":"ContainerDied","Data":"bfae7bf4ae9ee16b9ae5cd960c033ac74f28b45517b649957a0c2746baa1f61e"}
Jul 28 00:50:52.933136 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.932829    2576 scope.go:117] "RemoveContainer" containerID="3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598"
Jul 28 00:50:52.940712 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.940696    2576 scope.go:117] "RemoveContainer" containerID="3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598"
Jul 28 00:50:52.940953 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:50:52.940934    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598\": container with ID starting with 3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598 not found: ID does not exist" containerID="3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598"
Jul 28 00:50:52.940994 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.940961    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598"} err="failed to get container status \"3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598\": rpc error: code = NotFound desc = could not find container \"3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598\": container with ID starting with 3d810c1e96e23146932f5d7f685a20edbc6e1bb1fb86ea0e3979eeaa5eb48598 not found: ID does not exist"
Jul 28 00:50:52.953044 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.953022    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58888d4f95-xj9nb"]
Jul 28 00:50:52.956050 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:52.956032    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58888d4f95-xj9nb"]
Jul 28 00:50:53.749533 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:50:53.749503    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1461d0-3bd7-42fe-aebc-c6f8769285a3" path="/var/lib/kubelet/pods/fe1461d0-3bd7-42fe-aebc-c6f8769285a3/volumes"
Jul 28 00:51:18.475258 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:18.475162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:19.715893 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:19.715847    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:35.601846 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:35.601814    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 00:51:40.053454 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:40.053418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:40.839635 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:40.839604    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:43.665799 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:43.665759    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:43.762096 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:43.762065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:45.375124 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:45.375095    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:45.482283 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:45.482250    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:45.595704 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:45.595669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:45.709245 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:45.709150    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:47.383838 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:47.383808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:47.772362 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:47.772288    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:50.449006 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:50.448976    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:50.506233 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:50.506200    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:53.752049 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.752015    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sr7kl"]
Jul 28 00:51:53.753061 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.753037    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1461d0-3bd7-42fe-aebc-c6f8769285a3" containerName="console"
Jul 28 00:51:53.753196 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.753185    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1461d0-3bd7-42fe-aebc-c6f8769285a3" containerName="console"
Jul 28 00:51:53.753464 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.753451    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1461d0-3bd7-42fe-aebc-c6f8769285a3" containerName="console"
Jul 28 00:51:53.756865 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.756843    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:53.759726 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.759705    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 00:51:53.759842 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.759790    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 00:51:53.760888 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.760873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-blthb\""
Jul 28 00:51:53.763499 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.763353    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sr7kl"]
Jul 28 00:51:53.874629 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.874589    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855tv\" (UniqueName: \"kubernetes.io/projected/eae9e421-6cdb-4bde-b963-e0622dd8e10e-kube-api-access-855tv\") pod \"cert-manager-webhook-6486ffd997-sr7kl\" (UID: \"eae9e421-6cdb-4bde-b963-e0622dd8e10e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:53.874885 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.874653    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae9e421-6cdb-4bde-b963-e0622dd8e10e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sr7kl\" (UID: \"eae9e421-6cdb-4bde-b963-e0622dd8e10e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:53.975416 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.975383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855tv\" (UniqueName: \"kubernetes.io/projected/eae9e421-6cdb-4bde-b963-e0622dd8e10e-kube-api-access-855tv\") pod \"cert-manager-webhook-6486ffd997-sr7kl\" (UID: \"eae9e421-6cdb-4bde-b963-e0622dd8e10e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:53.975568 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.975422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae9e421-6cdb-4bde-b963-e0622dd8e10e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sr7kl\" (UID: \"eae9e421-6cdb-4bde-b963-e0622dd8e10e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:53.984021 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.983981    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eae9e421-6cdb-4bde-b963-e0622dd8e10e-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-sr7kl\" (UID: \"eae9e421-6cdb-4bde-b963-e0622dd8e10e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:53.984144 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:53.984082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855tv\" (UniqueName: \"kubernetes.io/projected/eae9e421-6cdb-4bde-b963-e0622dd8e10e-kube-api-access-855tv\") pod \"cert-manager-webhook-6486ffd997-sr7kl\" (UID: \"eae9e421-6cdb-4bde-b963-e0622dd8e10e\") " pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:54.074102 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:54.074066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:54.195592 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:54.195558    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-sr7kl"]
Jul 28 00:51:54.198131 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:51:54.198102    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeae9e421_6cdb_4bde_b963_e0622dd8e10e.slice/crio-a461e0dbf82aba1947a189304d57f884cd0955d6e24a85755c7a84ca1dcca09a WatchSource:0}: Error finding container a461e0dbf82aba1947a189304d57f884cd0955d6e24a85755c7a84ca1dcca09a: Status 404 returned error can't find the container with id a461e0dbf82aba1947a189304d57f884cd0955d6e24a85755c7a84ca1dcca09a
Jul 28 00:51:54.199981 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:54.199964    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 00:51:55.167654 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:55.167594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl" event={"ID":"eae9e421-6cdb-4bde-b963-e0622dd8e10e","Type":"ContainerStarted","Data":"a461e0dbf82aba1947a189304d57f884cd0955d6e24a85755c7a84ca1dcca09a"}
Jul 28 00:51:55.576791 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:55.576758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:57.176650 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:57.176618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl" event={"ID":"eae9e421-6cdb-4bde-b963-e0622dd8e10e","Type":"ContainerStarted","Data":"5b38c3975561a16a9c0d27baca67cea6a313f7dea3d51698cf9d2c005085bc1e"}
Jul 28 00:51:57.177035 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:57.176681    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:51:57.285242 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:57.285188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:57.385578 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:57.385548    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:58.119573 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:58.119532    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:51:58.219212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:51:58.219172    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:02.953769 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:02.953732    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:03.060496 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:03.060473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:03.183443 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:03.183416    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl"
Jul 28 00:52:03.202046 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:03.202000    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-sr7kl" podStartSLOduration=7.360707406 podStartE2EDuration="10.201986382s" totalImagesPullingTime="2.841278976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:51:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:51:57.197894032 +0000 UTC m=+321.947032757" watchObservedRunningTime="2026-07-28 00:52:03.201986382 +0000 UTC m=+327.951125106"
Jul 28 00:52:04.865271 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:04.865236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:21.698144 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:21.698108    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:21.796583 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:21.796547    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:22.316478 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:22.316448    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:22.877402 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:22.877363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:22.976595 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:22.976565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.018878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.018848    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"]
Jul 28 00:52:23.025832 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.025810    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.030143 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.030115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s5btk\""
Jul 28 00:52:23.030268 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.030198    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 00:52:23.030346 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.030304    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 00:52:23.030346 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.030314    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 00:52:23.030591 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.030576    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 00:52:23.045261 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.045242    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"]
Jul 28 00:52:23.170505 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.170379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.212085 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.212062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs4fz\" (UniqueName: \"kubernetes.io/projected/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-kube-api-access-gs4fz\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.212248 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.212105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.212248 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.212141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.212248 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.212173    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.212360 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.212264    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.274971 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.274932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.312789 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.312764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs4fz\" (UniqueName: \"kubernetes.io/projected/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-kube-api-access-gs4fz\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.312875 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.312807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.312875 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.312859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.312968 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.312877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.312968 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.312925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.313420 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.313395    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-manifests\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.315111 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.315088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-tmp\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.315361 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.315343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-webhook-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.315495 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.315477    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-apiservice-cert\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.321258 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.321237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs4fz\" (UniqueName: \"kubernetes.io/projected/e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050-kube-api-access-gs4fz\") pod \"opendatahub-operator-controller-manager-644f76477d-mtj7j\" (UID: \"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050\") " pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.335195 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.335177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:23.369973 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.369938    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.493108 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.493063    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"]
Jul 28 00:52:23.495732 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.495698    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.495882 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:52:23.495865    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1517e3f_f9a8_4d3e_b1d3_3d4f92ba4050.slice/crio-64b4040c2fba08f7480e4ee89d22382af17d0440dd6bfeed674a28b2c504cd93 WatchSource:0}: Error finding container 64b4040c2fba08f7480e4ee89d22382af17d0440dd6bfeed674a28b2c504cd93: Status 404 returned error can't find the container with id 64b4040c2fba08f7480e4ee89d22382af17d0440dd6bfeed674a28b2c504cd93
Jul 28 00:52:23.567639 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.567599    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.728603 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.728568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:23.831508 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:23.831484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:24.271011 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:24.270932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j" event={"ID":"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050","Type":"ContainerStarted","Data":"64b4040c2fba08f7480e4ee89d22382af17d0440dd6bfeed674a28b2c504cd93"}
Jul 28 00:52:27.284353 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:27.284318    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j" event={"ID":"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050","Type":"ContainerStarted","Data":"fcdb942c0e4627e473be8a8765134fc418c0ea5726219c42acb78e6db4910445"}
Jul 28 00:52:28.290181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:28.290102    2576 generic.go:356] "Generic (PLEG): container finished" podID="e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050" containerID="fcdb942c0e4627e473be8a8765134fc418c0ea5726219c42acb78e6db4910445" exitCode=0
Jul 28 00:52:28.290181 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:28.290153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j" event={"ID":"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050","Type":"ContainerDied","Data":"fcdb942c0e4627e473be8a8765134fc418c0ea5726219c42acb78e6db4910445"}
Jul 28 00:52:29.176873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:29.176842    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:29.296143 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:29.296107    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j" event={"ID":"e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050","Type":"ContainerStarted","Data":"877d046c356aba03341a9c809f73549c3270c0fe6c0df9d921f42f471774b599"}
Jul 28 00:52:29.296527 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:29.296202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:29.301410 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:29.301390    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:29.337514 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:29.337471    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j" podStartSLOduration=2.957180813 podStartE2EDuration="7.337455583s" totalImagesPullingTime="3.304904906s" totalInitContainerRuntime="1.075369864s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:29.336586043 +0000 UTC m=+354.085724777" watchObservedRunningTime="2026-07-28 00:52:29.337455583 +0000 UTC m=+354.086594307"
Jul 28 00:52:32.737995 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:32.737963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:32.834726 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:32.834696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:34.334241 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:34.334198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:34.439387 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:34.439354    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:37.133927 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:37.133896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:37.241821 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:37.241783    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:40.302032 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:40.301995    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-644f76477d-mtj7j"
Jul 28 00:52:41.914093 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:41.914052    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:42.046599 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:42.046572    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:42.879282 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:42.879255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:42.979447 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:42.979417    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:43.079763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:43.079729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:43.182603 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:43.182523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:43.581484 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:43.581451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:43.719078 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:43.719048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:43.831120 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:43.831089    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:49.282345 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.282311    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"]
Jul 28 00:52:49.290749 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.290724    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.293460 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.293433    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 00:52:49.293460 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.293435    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 00:52:49.293642 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.293435    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 00:52:49.293642 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.293487    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 00:52:49.294883 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.294864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 00:52:49.294997 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.294923    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-24vfz\""
Jul 28 00:52:49.298426 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.298404    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"]
Jul 28 00:52:49.309596 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.309573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e8d2ecc-bba7-4358-8530-bc9045a945eb-manager-config\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.309687 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.309618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p54n\" (UniqueName: \"kubernetes.io/projected/6e8d2ecc-bba7-4358-8530-bc9045a945eb-kube-api-access-2p54n\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.309687 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.309676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e8d2ecc-bba7-4358-8530-bc9045a945eb-metrics-cert\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.309798 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.309705    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e8d2ecc-bba7-4358-8530-bc9045a945eb-cert\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.410867 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.410840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e8d2ecc-bba7-4358-8530-bc9045a945eb-manager-config\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.410987 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.410872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p54n\" (UniqueName: \"kubernetes.io/projected/6e8d2ecc-bba7-4358-8530-bc9045a945eb-kube-api-access-2p54n\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.410987 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.410903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e8d2ecc-bba7-4358-8530-bc9045a945eb-metrics-cert\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.411077 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.411014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e8d2ecc-bba7-4358-8530-bc9045a945eb-cert\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.411580 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.411558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e8d2ecc-bba7-4358-8530-bc9045a945eb-manager-config\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.413336 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.413313    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e8d2ecc-bba7-4358-8530-bc9045a945eb-metrics-cert\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.413416 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.413377    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e8d2ecc-bba7-4358-8530-bc9045a945eb-cert\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.418705 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.418679    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p54n\" (UniqueName: \"kubernetes.io/projected/6e8d2ecc-bba7-4358-8530-bc9045a945eb-kube-api-access-2p54n\") pod \"lws-controller-manager-f79f7c78b-kx45j\" (UID: \"6e8d2ecc-bba7-4358-8530-bc9045a945eb\") " pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.601444 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.601408    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:49.739681 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:49.739634    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"]
Jul 28 00:52:49.742600 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:52:49.742574    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8d2ecc_bba7_4358_8530_bc9045a945eb.slice/crio-7386b8b54dfb014c77c5e35b530b07c0a75cae5f2d173492e6ebae96d285c1e1 WatchSource:0}: Error finding container 7386b8b54dfb014c77c5e35b530b07c0a75cae5f2d173492e6ebae96d285c1e1: Status 404 returned error can't find the container with id 7386b8b54dfb014c77c5e35b530b07c0a75cae5f2d173492e6ebae96d285c1e1
Jul 28 00:52:50.370814 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:50.370781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j" event={"ID":"6e8d2ecc-bba7-4358-8530-bc9045a945eb","Type":"ContainerStarted","Data":"7386b8b54dfb014c77c5e35b530b07c0a75cae5f2d173492e6ebae96d285c1e1"}
Jul 28 00:52:51.983788 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:51.983754    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:52.039128 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.039100    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"]
Jul 28 00:52:52.049680 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.049655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.052695 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.052672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-k5hc5\""
Jul 28 00:52:52.052866 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.052672    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 28 00:52:52.059965 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.059938    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"]
Jul 28 00:52:52.064254 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.064191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:52.135767 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.135737    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-925hg\" (UniqueName: \"kubernetes.io/projected/cf6ef881-3c12-45bd-b734-a5e2833988e9-kube-api-access-925hg\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.135929 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.135849    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6ef881-3c12-45bd-b734-a5e2833988e9-cert\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.237115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.237030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6ef881-3c12-45bd-b734-a5e2833988e9-cert\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.237115 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.237107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-925hg\" (UniqueName: \"kubernetes.io/projected/cf6ef881-3c12-45bd-b734-a5e2833988e9-kube-api-access-925hg\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.237368 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:52:52.237201    2576 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 28 00:52:52.237368 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:52:52.237298    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf6ef881-3c12-45bd-b734-a5e2833988e9-cert podName:cf6ef881-3c12-45bd-b734-a5e2833988e9 nodeName:}" failed. No retries permitted until 2026-07-28 00:52:52.737274532 +0000 UTC m=+377.486413242 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/cf6ef881-3c12-45bd-b734-a5e2833988e9-cert") pod "kserve-controller-manager-d6f4b65b8-rxsnd" (UID: "cf6ef881-3c12-45bd-b734-a5e2833988e9") : secret "kserve-webhook-server-cert" not found
Jul 28 00:52:52.265928 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.265901    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-925hg\" (UniqueName: \"kubernetes.io/projected/cf6ef881-3c12-45bd-b734-a5e2833988e9-kube-api-access-925hg\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.405642 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.405606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:52.629391 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.629362    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:52.742175 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.742141    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6ef881-3c12-45bd-b734-a5e2833988e9-cert\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.744632 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.744609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cf6ef881-3c12-45bd-b734-a5e2833988e9-cert\") pod \"kserve-controller-manager-d6f4b65b8-rxsnd\" (UID: \"cf6ef881-3c12-45bd-b734-a5e2833988e9\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:52.963002 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:52.962975    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:53.107714 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.107682    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"]
Jul 28 00:52:53.109752 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:52:53.109726    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6ef881_3c12_45bd_b734_a5e2833988e9.slice/crio-4575cde502966a86f7043390fd6340c7f116dee80f57a8967cce9bcbb5053e0f WatchSource:0}: Error finding container 4575cde502966a86f7043390fd6340c7f116dee80f57a8967cce9bcbb5053e0f: Status 404 returned error can't find the container with id 4575cde502966a86f7043390fd6340c7f116dee80f57a8967cce9bcbb5053e0f
Jul 28 00:52:53.285688 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.285603    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:53.300148 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.300120    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"]
Jul 28 00:52:53.304923 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.304901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.308489 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.308467    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 00:52:53.308595 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.308513    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 00:52:53.309290 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.309269    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-kkjr4\""
Jul 28 00:52:53.309381 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.309319    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 00:52:53.311160 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.311140    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 00:52:53.324511 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.324493    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"]
Jul 28 00:52:53.347677 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.347657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625d301d-b05b-47f3-8fa5-c808343fa5b3-tls-certs\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.347752 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.347687    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/625d301d-b05b-47f3-8fa5-c808343fa5b3-tmp\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.347797 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.347754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-666ct\" (UniqueName: \"kubernetes.io/projected/625d301d-b05b-47f3-8fa5-c808343fa5b3-kube-api-access-666ct\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.384275 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.384251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j" event={"ID":"6e8d2ecc-bba7-4358-8530-bc9045a945eb","Type":"ContainerStarted","Data":"4920a23123e39e34303b5addd94e8764e25fb94b7341bf007bfb92db5361c6b7"}
Jul 28 00:52:53.384423 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.384397    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:52:53.385502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.385481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd" event={"ID":"cf6ef881-3c12-45bd-b734-a5e2833988e9","Type":"ContainerStarted","Data":"4575cde502966a86f7043390fd6340c7f116dee80f57a8967cce9bcbb5053e0f"}
Jul 28 00:52:53.421306 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.421268    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j" podStartSLOduration=1.6220765400000001 podStartE2EDuration="4.421255625s" totalImagesPullingTime="2.799179085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:53.420583188 +0000 UTC m=+378.169721924" watchObservedRunningTime="2026-07-28 00:52:53.421255625 +0000 UTC m=+378.170394348"
Jul 28 00:52:53.448393 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.448372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625d301d-b05b-47f3-8fa5-c808343fa5b3-tls-certs\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.448478 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.448401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/625d301d-b05b-47f3-8fa5-c808343fa5b3-tmp\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.448594 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:52:53.448579    2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 00:52:53.448641 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:52:53.448633    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/625d301d-b05b-47f3-8fa5-c808343fa5b3-tls-certs podName:625d301d-b05b-47f3-8fa5-c808343fa5b3 nodeName:}" failed. No retries permitted until 2026-07-28 00:52:53.948617509 +0000 UTC m=+378.697756211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/625d301d-b05b-47f3-8fa5-c808343fa5b3-tls-certs") pod "kube-auth-proxy-c44798b85-jzsjp" (UID: "625d301d-b05b-47f3-8fa5-c808343fa5b3") : secret "kube-auth-proxy-tls" not found
Jul 28 00:52:53.448738 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.448718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-666ct\" (UniqueName: \"kubernetes.io/projected/625d301d-b05b-47f3-8fa5-c808343fa5b3-kube-api-access-666ct\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.450541 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.450517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/625d301d-b05b-47f3-8fa5-c808343fa5b3-tmp\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.456002 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.455952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:53.460592 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.460574    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-666ct\" (UniqueName: \"kubernetes.io/projected/625d301d-b05b-47f3-8fa5-c808343fa5b3-kube-api-access-666ct\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.581576 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.581550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:53.665389 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.665362    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:53.953429 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.953342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625d301d-b05b-47f3-8fa5-c808343fa5b3-tls-certs\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:53.956072 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:53.956037    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/625d301d-b05b-47f3-8fa5-c808343fa5b3-tls-certs\") pod \"kube-auth-proxy-c44798b85-jzsjp\" (UID: \"625d301d-b05b-47f3-8fa5-c808343fa5b3\") " pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:54.214933 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:54.214853    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"
Jul 28 00:52:54.344465 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:54.344417    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:54.344662 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:54.344625    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-c44798b85-jzsjp"]
Jul 28 00:52:54.347580 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:52:54.347554    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod625d301d_b05b_47f3_8fa5_c808343fa5b3.slice/crio-cd1418c58721457966fa2d0bd9722a8ed9a8359b5d58de7bd03cd7c937059376 WatchSource:0}: Error finding container cd1418c58721457966fa2d0bd9722a8ed9a8359b5d58de7bd03cd7c937059376: Status 404 returned error can't find the container with id cd1418c58721457966fa2d0bd9722a8ed9a8359b5d58de7bd03cd7c937059376
Jul 28 00:52:54.390764 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:54.390737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp" event={"ID":"625d301d-b05b-47f3-8fa5-c808343fa5b3","Type":"ContainerStarted","Data":"cd1418c58721457966fa2d0bd9722a8ed9a8359b5d58de7bd03cd7c937059376"}
Jul 28 00:52:54.451553 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:54.451519    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:55.076582 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:55.076538    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:55.174724 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:55.174689    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:55.878806 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:55.878726    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:55.964265 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:55.964232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:56.358995 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:56.358963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:52:56.402942 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:56.402902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd" event={"ID":"cf6ef881-3c12-45bd-b734-a5e2833988e9","Type":"ContainerStarted","Data":"79265c87d1762576767560a14094b924e581d324bd4cf5b48d6da817093a6c6a"}
Jul 28 00:52:56.403666 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:56.403193    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:52:56.420504 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:56.420455    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd" podStartSLOduration=1.964671885 podStartE2EDuration="4.420441911s" totalImagesPullingTime="2.455770026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:56.418728541 +0000 UTC m=+381.167867265" watchObservedRunningTime="2026-07-28 00:52:56.420441911 +0000 UTC m=+381.169580635"
Jul 28 00:52:57.407912 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:57.407874    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp" event={"ID":"625d301d-b05b-47f3-8fa5-c808343fa5b3","Type":"ContainerStarted","Data":"639c2a4ea916bebb15b07be10bfc21ffd150ca217683dcd49b1c2b866b4f4d05"}
Jul 28 00:52:57.425381 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:57.425332    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-c44798b85-jzsjp" podStartSLOduration=1.750151844 podStartE2EDuration="4.425317949s" totalImagesPullingTime="2.675166105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:52:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:52:57.423254305 +0000 UTC m=+382.172393025" watchObservedRunningTime="2026-07-28 00:52:57.425317949 +0000 UTC m=+382.174456664"
Jul 28 00:52:59.971967 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:52:59.971928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:00.074019 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:00.073949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:00.627849 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:00.627818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:00.829090 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:00.829054    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:01.078296 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:01.078265    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:04.392866 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:04.392835    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f79f7c78b-kx45j"
Jul 28 00:53:15.204661 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:15.204615    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:27.413145 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:27.413115    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-rxsnd"
Jul 28 00:53:41.374906 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:41.374872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:43.354383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:43.354349    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:43.446532 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:43.446500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:43.516353 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:43.516324    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:44.276912 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:44.276882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:53:44.475634 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:53:44.475596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:12.285703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:12.285628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:13.012712 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:13.012681    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:14.199878 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:14.199848    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:14.374361 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:14.374320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:16.849808 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:16.849778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:17.002501 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.002462    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6644d58899-bzjfq"]
Jul 28 00:54:17.006258 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.006213    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.024734 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.024701    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6644d58899-bzjfq"]
Jul 28 00:54:17.098177 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.098144    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:17.129527 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-oauth-serving-cert\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.129527 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hss2h\" (UniqueName: \"kubernetes.io/projected/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-kube-api-access-hss2h\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.129527 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129513    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-config\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.129742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129623    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-oauth-config\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.129742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-trusted-ca-bundle\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.129742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129712    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-serving-cert\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.129742 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.129734    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-service-ca\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230406 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-oauth-serving-cert\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230510 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hss2h\" (UniqueName: \"kubernetes.io/projected/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-kube-api-access-hss2h\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230510 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-config\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230510 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-oauth-config\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-trusted-ca-bundle\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-serving-cert\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.230656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.230602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-service-ca\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.231246 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.231196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-oauth-serving-cert\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.231328 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.231200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-config\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.231328 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.231268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-service-ca\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.231586 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.231563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-trusted-ca-bundle\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.232838 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.232813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-oauth-config\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.233168 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.233146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-console-serving-cert\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.245826 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.245802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hss2h\" (UniqueName: \"kubernetes.io/projected/b58d20e0-bedd-45ca-a0ad-baa2893b0f42-kube-api-access-hss2h\") pod \"console-6644d58899-bzjfq\" (UID: \"b58d20e0-bedd-45ca-a0ad-baa2893b0f42\") " pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.247700 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.247679    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:17.317848 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.317824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:17.453722 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:54:17.453694    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58d20e0_bedd_45ca_a0ad_baa2893b0f42.slice/crio-616b47477663d165e37e88ab1356ec9812a576a731baddd47c68860ba7b5e2e1 WatchSource:0}: Error finding container 616b47477663d165e37e88ab1356ec9812a576a731baddd47c68860ba7b5e2e1: Status 404 returned error can't find the container with id 616b47477663d165e37e88ab1356ec9812a576a731baddd47c68860ba7b5e2e1
Jul 28 00:54:17.454948 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.454928    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6644d58899-bzjfq"]
Jul 28 00:54:17.736822 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.736736    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6644d58899-bzjfq" event={"ID":"b58d20e0-bedd-45ca-a0ad-baa2893b0f42","Type":"ContainerStarted","Data":"3b079c2d9dc59e072a3b7c5e25670105519394b37bfd08b4a22578afb3a9acfc"}
Jul 28 00:54:17.736822 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.736781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6644d58899-bzjfq" event={"ID":"b58d20e0-bedd-45ca-a0ad-baa2893b0f42","Type":"ContainerStarted","Data":"616b47477663d165e37e88ab1356ec9812a576a731baddd47c68860ba7b5e2e1"}
Jul 28 00:54:17.801348 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:17.801304    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6644d58899-bzjfq" podStartSLOduration=1.8012887 podStartE2EDuration="1.8012887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:17.799821563 +0000 UTC m=+462.548960286" watchObservedRunningTime="2026-07-28 00:54:17.8012887 +0000 UTC m=+462.550427424"
Jul 28 00:54:21.305369 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:21.305334    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:22.210757 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:22.210694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:24.073907 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:24.073873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:24.277776 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:24.277737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:26.520666 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.520633    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:26.615975 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.615947    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"]
Jul 28 00:54:26.619177 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.619161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:26.622189 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.622165    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 00:54:26.622444 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.622429    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 00:54:26.623509 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.623493    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dv5qf\""
Jul 28 00:54:26.631238 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.631197    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"]
Jul 28 00:54:26.812259 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.812216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdwz\" (UniqueName: \"kubernetes.io/projected/0d0dde3b-77e1-414a-9c7c-0edccb969f7b-kube-api-access-7jdwz\") pod \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" (UID: \"0d0dde3b-77e1-414a-9c7c-0edccb969f7b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:26.913516 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.913484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdwz\" (UniqueName: \"kubernetes.io/projected/0d0dde3b-77e1-414a-9c7c-0edccb969f7b-kube-api-access-7jdwz\") pod \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" (UID: \"0d0dde3b-77e1-414a-9c7c-0edccb969f7b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:26.943111 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:26.943087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdwz\" (UniqueName: \"kubernetes.io/projected/0d0dde3b-77e1-414a-9c7c-0edccb969f7b-kube-api-access-7jdwz\") pod \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" (UID: \"0d0dde3b-77e1-414a-9c7c-0edccb969f7b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:27.230276 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.230202    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:27.319003 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.318698    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:27.319003 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.318806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:27.323413 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.323389    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:27.376300 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:54:27.376276    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0dde3b_77e1_414a_9c7c_0edccb969f7b.slice/crio-8755e3b7f986cae2abc9e9939e97748aea12c89064a2e57c0fd1b0086df9e904 WatchSource:0}: Error finding container 8755e3b7f986cae2abc9e9939e97748aea12c89064a2e57c0fd1b0086df9e904: Status 404 returned error can't find the container with id 8755e3b7f986cae2abc9e9939e97748aea12c89064a2e57c0fd1b0086df9e904
Jul 28 00:54:27.378057 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.378029    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"]
Jul 28 00:54:27.781291 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.781260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr" event={"ID":"0d0dde3b-77e1-414a-9c7c-0edccb969f7b","Type":"ContainerStarted","Data":"8755e3b7f986cae2abc9e9939e97748aea12c89064a2e57c0fd1b0086df9e904"}
Jul 28 00:54:27.785090 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.785064    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6644d58899-bzjfq"
Jul 28 00:54:27.841284 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.841258    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7f9b4999-zs5gb"]
Jul 28 00:54:27.985257 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:27.985215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:28.087383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:28.087355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:28.201187 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:28.201145    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:28.346008 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:28.345932    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:28.603191 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:28.603085    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:28.704052 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:28.704022    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:29.791832 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:29.791802    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr" event={"ID":"0d0dde3b-77e1-414a-9c7c-0edccb969f7b","Type":"ContainerStarted","Data":"f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2"}
Jul 28 00:54:29.792196 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:29.791862    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:29.813119 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:29.813071    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr" podStartSLOduration=1.9041974050000001 podStartE2EDuration="3.813056425s" totalImagesPullingTime="1.90885902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:29.811937734 +0000 UTC m=+474.561076461" watchObservedRunningTime="2026-07-28 00:54:29.813056425 +0000 UTC m=+474.562195148"
Jul 28 00:54:31.180327 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:31.180290    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:31.333118 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:31.333083    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:33.789763 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:33.789732    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:33.890715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:33.890688    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:40.242708 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:40.242658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:40.345198 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:40.345164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:40.798543 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:40.798517    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:41.925863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:41.925833    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"]
Jul 28 00:54:41.926270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:41.926001    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" containerName="manager" containerID="cri-o://f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2" gracePeriod=2
Jul 28 00:54:41.952743 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:41.952721    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"]
Jul 28 00:54:41.964069 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:41.964041    2576 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"0d0dde3b-77e1-414a-9c7c-0edccb969f7b\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-pz2zr\": pods \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:42.174200 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.174178    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:42.176943 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:42.176900    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:42.230354 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.230330    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jdwz\" (UniqueName: \"kubernetes.io/projected/0d0dde3b-77e1-414a-9c7c-0edccb969f7b-kube-api-access-7jdwz\") pod \"0d0dde3b-77e1-414a-9c7c-0edccb969f7b\" (UID: \"0d0dde3b-77e1-414a-9c7c-0edccb969f7b\") "
Jul 28 00:54:42.232311 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.232287    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0dde3b-77e1-414a-9c7c-0edccb969f7b-kube-api-access-7jdwz" (OuterVolumeSpecName: "kube-api-access-7jdwz") pod "0d0dde3b-77e1-414a-9c7c-0edccb969f7b" (UID: "0d0dde3b-77e1-414a-9c7c-0edccb969f7b"). InnerVolumeSpecName "kube-api-access-7jdwz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:54:42.331782 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.331759    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jdwz\" (UniqueName: \"kubernetes.io/projected/0d0dde3b-77e1-414a-9c7c-0edccb969f7b-kube-api-access-7jdwz\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:42.852570 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.852539    2576 generic.go:356] "Generic (PLEG): container finished" podID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" containerID="f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2" exitCode=0
Jul 28 00:54:42.852718 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.852582    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:42.852718 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.852602    2576 scope.go:117] "RemoveContainer" containerID="f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2"
Jul 28 00:54:42.855681 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:42.855651    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:42.860743 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.860723    2576 scope.go:117] "RemoveContainer" containerID="f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2"
Jul 28 00:54:42.861017 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:42.860998    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2\": container with ID starting with f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2 not found: ID does not exist" containerID="f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2"
Jul 28 00:54:42.861073 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:42.861024    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2"} err="failed to get container status \"f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2\": rpc error: code = NotFound desc = could not find container \"f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2\": container with ID starting with f092439906449f409298b5caf148367b64ffe9f0ed531ec7ca73a56510512ac2 not found: ID does not exist"
Jul 28 00:54:42.863350 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:42.863329    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:43.515569 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.515533    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"]
Jul 28 00:54:43.516030 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.515972    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" containerName="manager"
Jul 28 00:54:43.516030 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.515988    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" containerName="manager"
Jul 28 00:54:43.516102 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.516048    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" containerName="manager"
Jul 28 00:54:43.519980 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.519963    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.523716 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:43.523679    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pz2zr\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pz2zr"
Jul 28 00:54:43.524237 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.524209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 00:54:43.524345 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.524275    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 00:54:43.524345 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.524282    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b5q2g\""
Jul 28 00:54:43.540854 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.540830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ab07e3-97db-46b5-be1f-169e5ab12e42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qvh8v\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.540965 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.540883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4jd\" (UniqueName: \"kubernetes.io/projected/f0ab07e3-97db-46b5-be1f-169e5ab12e42-kube-api-access-6s4jd\") pod \"kuadrant-operator-controller-manager-86878c769f-qvh8v\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.542085 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.542062    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"]
Jul 28 00:54:43.641382 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.641358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ab07e3-97db-46b5-be1f-169e5ab12e42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qvh8v\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.641481 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.641404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4jd\" (UniqueName: \"kubernetes.io/projected/f0ab07e3-97db-46b5-be1f-169e5ab12e42-kube-api-access-6s4jd\") pod \"kuadrant-operator-controller-manager-86878c769f-qvh8v\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.641717 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.641699    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ab07e3-97db-46b5-be1f-169e5ab12e42-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-qvh8v\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.665470 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.665443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4jd\" (UniqueName: \"kubernetes.io/projected/f0ab07e3-97db-46b5-be1f-169e5ab12e42-kube-api-access-6s4jd\") pod \"kuadrant-operator-controller-manager-86878c769f-qvh8v\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.749366 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.749342    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d0dde3b-77e1-414a-9c7c-0edccb969f7b" path="/var/lib/kubelet/pods/0d0dde3b-77e1-414a-9c7c-0edccb969f7b/volumes"
Jul 28 00:54:43.830015 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.829985    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:43.983312 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:43.983198    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"]
Jul 28 00:54:43.986191 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:54:43.986162    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0ab07e3_97db_46b5_be1f_169e5ab12e42.slice/crio-4f44ffa2cd3ced8c111b4afa1c05b5ce2f085a8bfa71604c14f68da439a48f1e WatchSource:0}: Error finding container 4f44ffa2cd3ced8c111b4afa1c05b5ce2f085a8bfa71604c14f68da439a48f1e: Status 404 returned error can't find the container with id 4f44ffa2cd3ced8c111b4afa1c05b5ce2f085a8bfa71604c14f68da439a48f1e
Jul 28 00:54:44.862839 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:44.862800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v" event={"ID":"f0ab07e3-97db-46b5-be1f-169e5ab12e42","Type":"ContainerStarted","Data":"4f44ffa2cd3ced8c111b4afa1c05b5ce2f085a8bfa71604c14f68da439a48f1e"}
Jul 28 00:54:45.001277 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:45.001246    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:45.113376 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:45.113297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:48.807431 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:48.807396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:48.882881 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:48.882850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v" event={"ID":"f0ab07e3-97db-46b5-be1f-169e5ab12e42","Type":"ContainerStarted","Data":"2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0"}
Jul 28 00:54:48.883081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:48.883033    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:54:48.902799 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:48.902755    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v" podStartSLOduration=1.734916463 podStartE2EDuration="5.90274196s" totalImagesPullingTime="4.167825497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:54:48.901325255 +0000 UTC m=+493.650463979" watchObservedRunningTime="2026-07-28 00:54:48.90274196 +0000 UTC m=+493.651880686"
Jul 28 00:54:48.904259 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:48.904238    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:49.198123 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:49.198050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:49.260410 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:49.260382    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:52.861015 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:52.860978    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5f7f9b4999-zs5gb" podUID="65489689-9fa0-43fe-82f7-b583ff91b1fb" containerName="console" containerID="cri-o://c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24" gracePeriod=15
Jul 28 00:54:53.106055 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.106031    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:54:53.216412 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216327    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-oauth-config\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216412 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216372    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdnxk\" (UniqueName: \"kubernetes.io/projected/65489689-9fa0-43fe-82f7-b583ff91b1fb-kube-api-access-cdnxk\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216412 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216396    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-oauth-serving-cert\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216412 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216411    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-service-ca\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216727 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216437    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-trusted-ca-bundle\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216727 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216453    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-config\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216727 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216504    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-serving-cert\") pod \"65489689-9fa0-43fe-82f7-b583ff91b1fb\" (UID: \"65489689-9fa0-43fe-82f7-b583ff91b1fb\") "
Jul 28 00:54:53.216946 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216867    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-service-ca" (OuterVolumeSpecName: "service-ca") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:54:53.216946 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216877    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:54:53.217044 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216952    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:54:53.217044 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.216942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-config" (OuterVolumeSpecName: "console-config") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 00:54:53.218587 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.218566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:54:53.218689 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.218623    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:54:53.218735 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.218722    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65489689-9fa0-43fe-82f7-b583ff91b1fb-kube-api-access-cdnxk" (OuterVolumeSpecName: "kube-api-access-cdnxk") pod "65489689-9fa0-43fe-82f7-b583ff91b1fb" (UID: "65489689-9fa0-43fe-82f7-b583ff91b1fb"). InnerVolumeSpecName "kube-api-access-cdnxk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:54:53.222050 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.222026    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:53.291045 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.291016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:53.317581 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317561    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.317656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317581    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-oauth-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.317656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317591    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdnxk\" (UniqueName: \"kubernetes.io/projected/65489689-9fa0-43fe-82f7-b583ff91b1fb-kube-api-access-cdnxk\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.317656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317600    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-oauth-serving-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.317656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317610    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.317656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317618    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-trusted-ca-bundle\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.317656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.317627    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/65489689-9fa0-43fe-82f7-b583ff91b1fb-console-config\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:54:53.905059 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.905022    2576 generic.go:356] "Generic (PLEG): container finished" podID="65489689-9fa0-43fe-82f7-b583ff91b1fb" containerID="c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24" exitCode=0
Jul 28 00:54:53.905502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.905071    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f7f9b4999-zs5gb"
Jul 28 00:54:53.905502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.905073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f9b4999-zs5gb" event={"ID":"65489689-9fa0-43fe-82f7-b583ff91b1fb","Type":"ContainerDied","Data":"c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24"}
Jul 28 00:54:53.905502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.905147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f7f9b4999-zs5gb" event={"ID":"65489689-9fa0-43fe-82f7-b583ff91b1fb","Type":"ContainerDied","Data":"fd53380333ef91b6dce1ba056d7b04bc37e65dd458ffbe129d1cfe0f61640a5c"}
Jul 28 00:54:53.905502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.905179    2576 scope.go:117] "RemoveContainer" containerID="c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24"
Jul 28 00:54:53.913104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.913081    2576 scope.go:117] "RemoveContainer" containerID="c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24"
Jul 28 00:54:53.913377 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:54:53.913359    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24\": container with ID starting with c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24 not found: ID does not exist" containerID="c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24"
Jul 28 00:54:53.913446 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.913386    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24"} err="failed to get container status \"c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24\": rpc error: code = NotFound desc = could not find container \"c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24\": container with ID starting with c1e61704870d2c97e8596d5a1cb801832221b20e0b357bf2e87b4a1b20047d24 not found: ID does not exist"
Jul 28 00:54:53.924202 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.924181    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f7f9b4999-zs5gb"]
Jul 28 00:54:53.926845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:53.926825    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f7f9b4999-zs5gb"]
Jul 28 00:54:55.756444 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:55.756410    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65489689-9fa0-43fe-82f7-b583ff91b1fb" path="/var/lib/kubelet/pods/65489689-9fa0-43fe-82f7-b583ff91b1fb/volumes"
Jul 28 00:54:59.244130 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:59.244104    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:59.354777 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:59.354746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:54:59.889243 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:54:59.889197    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 00:55:03.999252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:03.999208    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:04.100533 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:04.100500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:15.917823 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:15.917787    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:16.267212 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:16.267140    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:16.378327 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:16.378294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:19.387922 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:19.387886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:19.469797 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:19.469763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:19.527869 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:19.527837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:20.146496 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.146463    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-rjwpr"]
Jul 28 00:55:20.146765 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.146753    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65489689-9fa0-43fe-82f7-b583ff91b1fb" containerName="console"
Jul 28 00:55:20.146810 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.146766    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="65489689-9fa0-43fe-82f7-b583ff91b1fb" containerName="console"
Jul 28 00:55:20.146845 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.146825    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="65489689-9fa0-43fe-82f7-b583ff91b1fb" containerName="console"
Jul 28 00:55:20.177626 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.177591    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rjwpr"]
Jul 28 00:55:20.177787 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.177690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:20.180178 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.180152    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gfzvw\""
Jul 28 00:55:20.180427 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.180406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:20.280613 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.280577    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:20.325444 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.325411    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtpn9\" (UniqueName: \"kubernetes.io/projected/c085e841-11e4-4762-bc4b-281f38a27957-kube-api-access-jtpn9\") pod \"authorino-58cd955675-rjwpr\" (UID: \"c085e841-11e4-4762-bc4b-281f38a27957\") " pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:20.426591 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.426517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtpn9\" (UniqueName: \"kubernetes.io/projected/c085e841-11e4-4762-bc4b-281f38a27957-kube-api-access-jtpn9\") pod \"authorino-58cd955675-rjwpr\" (UID: \"c085e841-11e4-4762-bc4b-281f38a27957\") " pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:20.434717 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.434681    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtpn9\" (UniqueName: \"kubernetes.io/projected/c085e841-11e4-4762-bc4b-281f38a27957-kube-api-access-jtpn9\") pod \"authorino-58cd955675-rjwpr\" (UID: \"c085e841-11e4-4762-bc4b-281f38a27957\") " pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:20.486401 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.486367    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:20.624540 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:20.624513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-rjwpr"]
Jul 28 00:55:20.626273 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:55:20.626244    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc085e841_11e4_4762_bc4b_281f38a27957.slice/crio-fb8ac5ee8682d12d2710efff21e7e0597dad025713b382fc6ae3ce49ff6a2eb7 WatchSource:0}: Error finding container fb8ac5ee8682d12d2710efff21e7e0597dad025713b382fc6ae3ce49ff6a2eb7: Status 404 returned error can't find the container with id fb8ac5ee8682d12d2710efff21e7e0597dad025713b382fc6ae3ce49ff6a2eb7
Jul 28 00:55:21.022563 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:21.022531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rjwpr" event={"ID":"c085e841-11e4-4762-bc4b-281f38a27957","Type":"ContainerStarted","Data":"fb8ac5ee8682d12d2710efff21e7e0597dad025713b382fc6ae3ce49ff6a2eb7"}
Jul 28 00:55:24.040216 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:24.040182    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rjwpr" event={"ID":"c085e841-11e4-4762-bc4b-281f38a27957","Type":"ContainerStarted","Data":"67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f"}
Jul 28 00:55:24.054773 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:24.054729    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-rjwpr" podStartSLOduration=0.786575123 podStartE2EDuration="4.054714205s" totalImagesPullingTime="3.268139082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:24.05356759 +0000 UTC m=+528.802706315" watchObservedRunningTime="2026-07-28 00:55:24.054714205 +0000 UTC m=+528.803852928"
Jul 28 00:55:24.288557 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:24.288482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:24.349257 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:24.349205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:25.419275 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:25.419240    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rjwpr"]
Jul 28 00:55:26.047972 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:26.047935    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-rjwpr" podUID="c085e841-11e4-4762-bc4b-281f38a27957" containerName="authorino" containerID="cri-o://67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f" gracePeriod=30
Jul 28 00:55:26.286483 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:26.286462    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:26.377157 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:26.377129    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtpn9\" (UniqueName: \"kubernetes.io/projected/c085e841-11e4-4762-bc4b-281f38a27957-kube-api-access-jtpn9\") pod \"c085e841-11e4-4762-bc4b-281f38a27957\" (UID: \"c085e841-11e4-4762-bc4b-281f38a27957\") "
Jul 28 00:55:26.379119 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:26.379093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c085e841-11e4-4762-bc4b-281f38a27957-kube-api-access-jtpn9" (OuterVolumeSpecName: "kube-api-access-jtpn9") pod "c085e841-11e4-4762-bc4b-281f38a27957" (UID: "c085e841-11e4-4762-bc4b-281f38a27957"). InnerVolumeSpecName "kube-api-access-jtpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:55:26.478104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:26.478081    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtpn9\" (UniqueName: \"kubernetes.io/projected/c085e841-11e4-4762-bc4b-281f38a27957-kube-api-access-jtpn9\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:55:27.052717 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.052686    2576 generic.go:356] "Generic (PLEG): container finished" podID="c085e841-11e4-4762-bc4b-281f38a27957" containerID="67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f" exitCode=0
Jul 28 00:55:27.052886 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.052735    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-rjwpr"
Jul 28 00:55:27.052886 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.052756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rjwpr" event={"ID":"c085e841-11e4-4762-bc4b-281f38a27957","Type":"ContainerDied","Data":"67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f"}
Jul 28 00:55:27.052886 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.052786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-rjwpr" event={"ID":"c085e841-11e4-4762-bc4b-281f38a27957","Type":"ContainerDied","Data":"fb8ac5ee8682d12d2710efff21e7e0597dad025713b382fc6ae3ce49ff6a2eb7"}
Jul 28 00:55:27.052886 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.052803    2576 scope.go:117] "RemoveContainer" containerID="67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f"
Jul 28 00:55:27.060950 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.060936    2576 scope.go:117] "RemoveContainer" containerID="67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f"
Jul 28 00:55:27.061188 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:55:27.061171    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f\": container with ID starting with 67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f not found: ID does not exist" containerID="67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f"
Jul 28 00:55:27.061249 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.061194    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f"} err="failed to get container status \"67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f\": rpc error: code = NotFound desc = could not find container \"67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f\": container with ID starting with 67ec52c095d07fec8f20a330840c3ca1f9e7c4a85d25ac05b9ca69660f19dc4f not found: ID does not exist"
Jul 28 00:55:27.071978 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.071956    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-rjwpr"]
Jul 28 00:55:27.073872 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.073852    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-rjwpr"]
Jul 28 00:55:27.750093 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:27.750055    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c085e841-11e4-4762-bc4b-281f38a27957" path="/var/lib/kubelet/pods/c085e841-11e4-4762-bc4b-281f38a27957/volumes"
Jul 28 00:55:30.103066 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:30.103019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:30.152048 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:30.152017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:35.391111 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:35.391042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:35.493331 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:35.493294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:35.850183 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:35.850135    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:47.816861 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:47.816825    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:47.992632 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:47.992604    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:48.038928 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.038904    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2zxhm"]
Jul 28 00:55:48.039198 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.039187    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c085e841-11e4-4762-bc4b-281f38a27957" containerName="authorino"
Jul 28 00:55:48.039263 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.039200    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c085e841-11e4-4762-bc4b-281f38a27957" containerName="authorino"
Jul 28 00:55:48.039302 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.039282    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c085e841-11e4-4762-bc4b-281f38a27957" containerName="authorino"
Jul 28 00:55:48.043387 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.043366    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:48.045906 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.045885    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gfzvw\""
Jul 28 00:55:48.048746 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.048726    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2zxhm"]
Jul 28 00:55:48.142920 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.142857    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fbb\" (UniqueName: \"kubernetes.io/projected/7e983f67-3eb6-4939-95b8-4b8eaa303d1c-kube-api-access-x9fbb\") pod \"authorino-5d4b67c446-2zxhm\" (UID: \"7e983f67-3eb6-4939-95b8-4b8eaa303d1c\") " pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:48.243341 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.243318    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fbb\" (UniqueName: \"kubernetes.io/projected/7e983f67-3eb6-4939-95b8-4b8eaa303d1c-kube-api-access-x9fbb\") pod \"authorino-5d4b67c446-2zxhm\" (UID: \"7e983f67-3eb6-4939-95b8-4b8eaa303d1c\") " pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:48.252139 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.252117    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fbb\" (UniqueName: \"kubernetes.io/projected/7e983f67-3eb6-4939-95b8-4b8eaa303d1c-kube-api-access-x9fbb\") pod \"authorino-5d4b67c446-2zxhm\" (UID: \"7e983f67-3eb6-4939-95b8-4b8eaa303d1c\") " pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:48.271533 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.271510    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2zxhm"]
Jul 28 00:55:48.271703 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.271692    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:48.290798 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.290774    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-ccbc7d556-frbpv"]
Jul 28 00:55:48.294502 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.294480    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:48.298355 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.298333    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-ccbc7d556-frbpv"]
Jul 28 00:55:48.344110 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.344087    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dscq7\" (UniqueName: \"kubernetes.io/projected/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de-kube-api-access-dscq7\") pod \"authorino-ccbc7d556-frbpv\" (UID: \"2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de\") " pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:48.379401 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.379370    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ccbc7d556-frbpv"]
Jul 28 00:55:48.379706 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:55:48.379677    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dscq7], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-ccbc7d556-frbpv" podUID="2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de"
Jul 28 00:55:48.405159 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.405103    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-798bf9d948-xzpxj"]
Jul 28 00:55:48.408506 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.408488    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2zxhm"]
Jul 28 00:55:48.408619 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.408570    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.408679 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:55:48.408634    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e983f67_3eb6_4939_95b8_4b8eaa303d1c.slice/crio-77b4602fc9321b3976f63cea2b2ec9eddb5e178bf3536021945952eda5e3e320 WatchSource:0}: Error finding container 77b4602fc9321b3976f63cea2b2ec9eddb5e178bf3536021945952eda5e3e320: Status 404 returned error can't find the container with id 77b4602fc9321b3976f63cea2b2ec9eddb5e178bf3536021945952eda5e3e320
Jul 28 00:55:48.411104 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.411088    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 00:55:48.414844 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.414825    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798bf9d948-xzpxj"]
Jul 28 00:55:48.444780 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.444762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/526164d7-acab-4bc5-8f82-eb6ffad2b692-tls-cert\") pod \"authorino-798bf9d948-xzpxj\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") " pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.444872 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.444823    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dscq7\" (UniqueName: \"kubernetes.io/projected/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de-kube-api-access-dscq7\") pod \"authorino-ccbc7d556-frbpv\" (UID: \"2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de\") " pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:48.444957 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.444879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvphq\" (UniqueName: \"kubernetes.io/projected/526164d7-acab-4bc5-8f82-eb6ffad2b692-kube-api-access-jvphq\") pod \"authorino-798bf9d948-xzpxj\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") " pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.452467 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.452440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dscq7\" (UniqueName: \"kubernetes.io/projected/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de-kube-api-access-dscq7\") pod \"authorino-ccbc7d556-frbpv\" (UID: \"2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de\") " pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:48.545265 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.545240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/526164d7-acab-4bc5-8f82-eb6ffad2b692-tls-cert\") pod \"authorino-798bf9d948-xzpxj\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") " pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.545400 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.545317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvphq\" (UniqueName: \"kubernetes.io/projected/526164d7-acab-4bc5-8f82-eb6ffad2b692-kube-api-access-jvphq\") pod \"authorino-798bf9d948-xzpxj\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") " pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.547559 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.547542    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/526164d7-acab-4bc5-8f82-eb6ffad2b692-tls-cert\") pod \"authorino-798bf9d948-xzpxj\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") " pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.552698 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.552675    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvphq\" (UniqueName: \"kubernetes.io/projected/526164d7-acab-4bc5-8f82-eb6ffad2b692-kube-api-access-jvphq\") pod \"authorino-798bf9d948-xzpxj\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") " pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.724081 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.724010    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:55:48.859826 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:48.859806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798bf9d948-xzpxj"]
Jul 28 00:55:48.862727 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:55:48.862699    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod526164d7_acab_4bc5_8f82_eb6ffad2b692.slice/crio-1d21b73d103fa0b0275c699380fe2007ebda2065a33c169738d24b2a0666c546 WatchSource:0}: Error finding container 1d21b73d103fa0b0275c699380fe2007ebda2065a33c169738d24b2a0666c546: Status 404 returned error can't find the container with id 1d21b73d103fa0b0275c699380fe2007ebda2065a33c169738d24b2a0666c546
Jul 28 00:55:49.143390 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.143354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2zxhm" event={"ID":"7e983f67-3eb6-4939-95b8-4b8eaa303d1c","Type":"ContainerStarted","Data":"bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9"}
Jul 28 00:55:49.143592 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.143400    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2zxhm" event={"ID":"7e983f67-3eb6-4939-95b8-4b8eaa303d1c","Type":"ContainerStarted","Data":"77b4602fc9321b3976f63cea2b2ec9eddb5e178bf3536021945952eda5e3e320"}
Jul 28 00:55:49.143592 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.143398    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-2zxhm" podUID="7e983f67-3eb6-4939-95b8-4b8eaa303d1c" containerName="authorino" containerID="cri-o://bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9" gracePeriod=30
Jul 28 00:55:49.144501 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.144475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:49.144591 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.144499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798bf9d948-xzpxj" event={"ID":"526164d7-acab-4bc5-8f82-eb6ffad2b692","Type":"ContainerStarted","Data":"1d21b73d103fa0b0275c699380fe2007ebda2065a33c169738d24b2a0666c546"}
Jul 28 00:55:49.151383 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.151364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:49.157477 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.157440    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-2zxhm" podStartSLOduration=0.835821273 podStartE2EDuration="1.157426701s" totalImagesPullingTime="321.605428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:49.156971995 +0000 UTC m=+553.906110747" watchObservedRunningTime="2026-07-28 00:55:49.157426701 +0000 UTC m=+553.906565424"
Jul 28 00:55:49.251255 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.251213    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dscq7\" (UniqueName: \"kubernetes.io/projected/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de-kube-api-access-dscq7\") pod \"2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de\" (UID: \"2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de\") "
Jul 28 00:55:49.252933 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.252910    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de-kube-api-access-dscq7" (OuterVolumeSpecName: "kube-api-access-dscq7") pod "2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de" (UID: "2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de"). InnerVolumeSpecName "kube-api-access-dscq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:55:49.352734 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.352702    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dscq7\" (UniqueName: \"kubernetes.io/projected/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de-kube-api-access-dscq7\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:55:49.400335 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.400276    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:49.453204 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.453180    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9fbb\" (UniqueName: \"kubernetes.io/projected/7e983f67-3eb6-4939-95b8-4b8eaa303d1c-kube-api-access-x9fbb\") pod \"7e983f67-3eb6-4939-95b8-4b8eaa303d1c\" (UID: \"7e983f67-3eb6-4939-95b8-4b8eaa303d1c\") "
Jul 28 00:55:49.455072 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.455046    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e983f67-3eb6-4939-95b8-4b8eaa303d1c-kube-api-access-x9fbb" (OuterVolumeSpecName: "kube-api-access-x9fbb") pod "7e983f67-3eb6-4939-95b8-4b8eaa303d1c" (UID: "7e983f67-3eb6-4939-95b8-4b8eaa303d1c"). InnerVolumeSpecName "kube-api-access-x9fbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:55:49.554475 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:49.554433    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9fbb\" (UniqueName: \"kubernetes.io/projected/7e983f67-3eb6-4939-95b8-4b8eaa303d1c-kube-api-access-x9fbb\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:55:50.155419 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.155387    2576 generic.go:356] "Generic (PLEG): container finished" podID="7e983f67-3eb6-4939-95b8-4b8eaa303d1c" containerID="bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9" exitCode=0
Jul 28 00:55:50.155873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.155449    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-2zxhm"
Jul 28 00:55:50.155873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.155463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2zxhm" event={"ID":"7e983f67-3eb6-4939-95b8-4b8eaa303d1c","Type":"ContainerDied","Data":"bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9"}
Jul 28 00:55:50.155873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.155502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-2zxhm" event={"ID":"7e983f67-3eb6-4939-95b8-4b8eaa303d1c","Type":"ContainerDied","Data":"77b4602fc9321b3976f63cea2b2ec9eddb5e178bf3536021945952eda5e3e320"}
Jul 28 00:55:50.155873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.155520    2576 scope.go:117] "RemoveContainer" containerID="bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9"
Jul 28 00:55:50.156979 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.156951    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798bf9d948-xzpxj" event={"ID":"526164d7-acab-4bc5-8f82-eb6ffad2b692","Type":"ContainerStarted","Data":"8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7"}
Jul 28 00:55:50.156979 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.156970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-ccbc7d556-frbpv"
Jul 28 00:55:50.163494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.163475    2576 scope.go:117] "RemoveContainer" containerID="bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9"
Jul 28 00:55:50.163760 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:55:50.163741    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9\": container with ID starting with bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9 not found: ID does not exist" containerID="bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9"
Jul 28 00:55:50.163825 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.163766    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9"} err="failed to get container status \"bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9\": rpc error: code = NotFound desc = could not find container \"bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9\": container with ID starting with bb8bd09e697a9f93b2c19e22b86331a55e451bd5386d0e3dbc6c67f9cb04ccc9 not found: ID does not exist"
Jul 28 00:55:50.175381 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.175341    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-798bf9d948-xzpxj" podStartSLOduration=1.809715768 podStartE2EDuration="2.175329813s" totalImagesPullingTime="365.614045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:55:50.173484773 +0000 UTC m=+554.922623497" watchObservedRunningTime="2026-07-28 00:55:50.175329813 +0000 UTC m=+554.924468537"
Jul 28 00:55:50.187248 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.187211    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2zxhm"]
Jul 28 00:55:50.189663 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.189644    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-2zxhm"]
Jul 28 00:55:50.215676 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.215646    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-ccbc7d556-frbpv"]
Jul 28 00:55:50.225993 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:50.225971    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-ccbc7d556-frbpv"]
Jul 28 00:55:51.749659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:51.749618    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de" path="/var/lib/kubelet/pods/2444bcd0-d1ba-4d5c-a9f8-9acf5f6c01de/volumes"
Jul 28 00:55:51.750149 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:51.749932    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e983f67-3eb6-4939-95b8-4b8eaa303d1c" path="/var/lib/kubelet/pods/7e983f67-3eb6-4939-95b8-4b8eaa303d1c/volumes"
Jul 28 00:55:54.705404 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:54.705372    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:54.802197 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:54.802170    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:54.884206 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:54.884176    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:54.940873 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:54.940847    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:58.192355 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:58.192320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:55:58.300623 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:55:58.300592    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:30.545696 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:30.545657    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:31.767715 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.767681    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vtbwh"]
Jul 28 00:56:31.768076 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.767978    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e983f67-3eb6-4939-95b8-4b8eaa303d1c" containerName="authorino"
Jul 28 00:56:31.768076 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.767988    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e983f67-3eb6-4939-95b8-4b8eaa303d1c" containerName="authorino"
Jul 28 00:56:31.768076 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.768050    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e983f67-3eb6-4939-95b8-4b8eaa303d1c" containerName="authorino"
Jul 28 00:56:31.770758 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.770737    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:31.773357 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.773332    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 00:56:31.773600 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.773585    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-g9rbm\""
Jul 28 00:56:31.779364 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.779340    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vtbwh"]
Jul 28 00:56:31.798189 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.798160    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:31.893431 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.893401    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7041fa74-1187-4ba0-aa28-1405298481c6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vtbwh\" (UID: \"7041fa74-1187-4ba0-aa28-1405298481c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:31.893579 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.893511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kkfr\" (UniqueName: \"kubernetes.io/projected/7041fa74-1187-4ba0-aa28-1405298481c6-kube-api-access-4kkfr\") pod \"payload-processing-5cc6cd444b-vtbwh\" (UID: \"7041fa74-1187-4ba0-aa28-1405298481c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:31.994529 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.994505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7041fa74-1187-4ba0-aa28-1405298481c6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vtbwh\" (UID: \"7041fa74-1187-4ba0-aa28-1405298481c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:31.994656 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.994550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kkfr\" (UniqueName: \"kubernetes.io/projected/7041fa74-1187-4ba0-aa28-1405298481c6-kube-api-access-4kkfr\") pod \"payload-processing-5cc6cd444b-vtbwh\" (UID: \"7041fa74-1187-4ba0-aa28-1405298481c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:31.995282 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:31.995264    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7041fa74-1187-4ba0-aa28-1405298481c6-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-vtbwh\" (UID: \"7041fa74-1187-4ba0-aa28-1405298481c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:32.002514 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:32.002492    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kkfr\" (UniqueName: \"kubernetes.io/projected/7041fa74-1187-4ba0-aa28-1405298481c6-kube-api-access-4kkfr\") pod \"payload-processing-5cc6cd444b-vtbwh\" (UID: \"7041fa74-1187-4ba0-aa28-1405298481c6\") " pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:32.082126 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:32.082102    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:32.209234 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:32.209197    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-vtbwh"]
Jul 28 00:56:32.211738 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:56:32.211710    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7041fa74_1187_4ba0_aa28_1405298481c6.slice/crio-1f59d28dfa8d793a1bfbdd0b09346549b54fd9fe9782c83163316c3db26b62f2 WatchSource:0}: Error finding container 1f59d28dfa8d793a1bfbdd0b09346549b54fd9fe9782c83163316c3db26b62f2: Status 404 returned error can't find the container with id 1f59d28dfa8d793a1bfbdd0b09346549b54fd9fe9782c83163316c3db26b62f2
Jul 28 00:56:32.337252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:32.337172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh" event={"ID":"7041fa74-1187-4ba0-aa28-1405298481c6","Type":"ContainerStarted","Data":"1f59d28dfa8d793a1bfbdd0b09346549b54fd9fe9782c83163316c3db26b62f2"}
Jul 28 00:56:34.348503 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:34.348470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh" event={"ID":"7041fa74-1187-4ba0-aa28-1405298481c6","Type":"ContainerStarted","Data":"e2d7ae9ab39cebc577856c009ed9007443ac6d9419fa979f8d4d224b6cac5d0d"}
Jul 28 00:56:34.349062 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:34.348598    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:34.367445 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:34.367399    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh" podStartSLOduration=1.900479494 podStartE2EDuration="3.367380082s" totalImagesPullingTime="1.466900588s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:56:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:56:34.366893956 +0000 UTC m=+599.116032680" watchObservedRunningTime="2026-07-28 00:56:34.367380082 +0000 UTC m=+599.116518807"
Jul 28 00:56:41.336927 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:41.336895    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:41.480232 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:41.480186    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:45.354791 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:45.354757    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-vtbwh"
Jul 28 00:56:51.202576 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:51.202545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:51.303531 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:51.303503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:56:53.758799 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:56:53.758768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:57:06.218895 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.218808    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"]
Jul 28 00:57:06.222370 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.222350    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.226187 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.226166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 28 00:57:06.226300 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.226167    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l9t28\""
Jul 28 00:57:06.226300 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.226167    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 00:57:06.226300 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.226186    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 00:57:06.232531 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.232509    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"]
Jul 28 00:57:06.323276 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.323242    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:57:06.357490 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.357467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4nr\" (UniqueName: \"kubernetes.io/projected/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-kube-api-access-xr4nr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.357582 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.357507    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.357630 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.357585    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.357630 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.357624    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.357728 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.357642    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.357780 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.357718    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.458934 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.458906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459054 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.458944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4nr\" (UniqueName: \"kubernetes.io/projected/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-kube-api-access-xr4nr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459054 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.458979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459054 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.459039    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459192 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.459079    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459192 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.459105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459420 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.459398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459494 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.459466    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.459550 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.459506    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.461256 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.461217    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.462108 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.462091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.466304 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.466281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4nr\" (UniqueName: \"kubernetes.io/projected/7eaad0c1-1a15-4729-b3d8-d59f43ab7d60-kube-api-access-xr4nr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-pxtw6\" (UID: \"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.533267 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.533199    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:06.658582 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.658550    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"]
Jul 28 00:57:06.661560 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:57:06.661535    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eaad0c1_1a15_4729_b3d8_d59f43ab7d60.slice/crio-1f3c08f778f0e2fb05198393e1608feff07d2c87b0f2c8a0b0fc517e43f2e339 WatchSource:0}: Error finding container 1f3c08f778f0e2fb05198393e1608feff07d2c87b0f2c8a0b0fc517e43f2e339: Status 404 returned error can't find the container with id 1f3c08f778f0e2fb05198393e1608feff07d2c87b0f2c8a0b0fc517e43f2e339
Jul 28 00:57:06.662798 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:06.662779    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 00:57:07.496414 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:07.496367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6" event={"ID":"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60","Type":"ContainerStarted","Data":"1f3c08f778f0e2fb05198393e1608feff07d2c87b0f2c8a0b0fc517e43f2e339"}
Jul 28 00:57:08.502589 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:08.502557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6" event={"ID":"7eaad0c1-1a15-4729-b3d8-d59f43ab7d60","Type":"ContainerStarted","Data":"1387ee4e093ddcacd8aae6b0144d400bf4040fa661b895f8344026cd8c4876e1"}
Jul 28 00:57:08.519481 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:08.519432    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6" podStartSLOduration=1.399381264 podStartE2EDuration="2.519403771s" totalImagesPullingTime="1.120022507s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:57:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:57:08.518807816 +0000 UTC m=+633.267946542" watchObservedRunningTime="2026-07-28 00:57:08.519403771 +0000 UTC m=+633.268542520"
Jul 28 00:57:16.533923 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:16.533892    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:16.534433 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:16.533950    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:16.546126 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:16.546099    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:16.855131 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:16.855099    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:57:17.552470 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:17.552435    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-pxtw6"
Jul 28 00:57:21.937439 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:21.937411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:57:29.715989 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.715956    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"]
Jul 28 00:57:29.719625 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.719604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.722186 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.722164    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 00:57:29.729626 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.729604    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"]
Jul 28 00:57:29.737270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.737245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:57:29.859061 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.859033    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.859211 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.859102    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.859211 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.859128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.859211 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.859191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpkbx\" (UniqueName: \"kubernetes.io/projected/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-kube-api-access-cpkbx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.859389 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.859248    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.859389 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.859310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960266 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960213    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960289    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960438 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960382    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960621 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpkbx\" (UniqueName: \"kubernetes.io/projected/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-kube-api-access-cpkbx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960621 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960621 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960603    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960778 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.960905 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.960886    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.962564 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.962538    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.962660 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.962646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:29.968237 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:29.968175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpkbx\" (UniqueName: \"kubernetes.io/projected/f247398a-2c9a-4ff3-a791-fbc8bbf7e882-kube-api-access-cpkbx\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-wmbxr\" (UID: \"f247398a-2c9a-4ff3-a791-fbc8bbf7e882\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:30.030078 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:30.030059    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:30.159213 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:30.159186    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"]
Jul 28 00:57:30.160965 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:57:30.160936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf247398a_2c9a_4ff3_a791_fbc8bbf7e882.slice/crio-92802d6c1c3ed494c57599b3bec36374f6d5e748cac83dcec0ebfaa1da2b2d2b WatchSource:0}: Error finding container 92802d6c1c3ed494c57599b3bec36374f6d5e748cac83dcec0ebfaa1da2b2d2b: Status 404 returned error can't find the container with id 92802d6c1c3ed494c57599b3bec36374f6d5e748cac83dcec0ebfaa1da2b2d2b
Jul 28 00:57:30.602534 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:30.602435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr" event={"ID":"f247398a-2c9a-4ff3-a791-fbc8bbf7e882","Type":"ContainerStarted","Data":"c9662acbab3d05059fab9bfe34afaadaedc299dcd011825ad467bfe83c1ee075"}
Jul 28 00:57:30.602534 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:30.602502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr" event={"ID":"f247398a-2c9a-4ff3-a791-fbc8bbf7e882","Type":"ContainerStarted","Data":"92802d6c1c3ed494c57599b3bec36374f6d5e748cac83dcec0ebfaa1da2b2d2b"}
Jul 28 00:57:30.618897 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:30.618845    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr" podStartSLOduration=1.436087952 podStartE2EDuration="1.618831949s" totalImagesPullingTime="182.743997ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:57:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:57:30.618274111 +0000 UTC m=+655.367412840" watchObservedRunningTime="2026-07-28 00:57:30.618831949 +0000 UTC m=+655.367970673"
Jul 28 00:57:37.418197 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.418166    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"]
Jul 28 00:57:37.421293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.421273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.423936 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.423918    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 00:57:37.430756 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.430734    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"]
Jul 28 00:57:37.444971 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.444951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:57:37.517036 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.517009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb49s\" (UniqueName: \"kubernetes.io/projected/bc67092a-61b5-44f5-b57a-6df58148de2e-kube-api-access-wb49s\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.517126 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.517041    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.517126 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.517062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.517126 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.517077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc67092a-61b5-44f5-b57a-6df58148de2e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.517270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.517132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.517270 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.517181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618542 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb49s\" (UniqueName: \"kubernetes.io/projected/bc67092a-61b5-44f5-b57a-6df58148de2e-kube-api-access-wb49s\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618547    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc67092a-61b5-44f5-b57a-6df58148de2e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618659 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618631    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618940 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618916    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.618991 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.618974    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.619063 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.619046    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.620948 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.620927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc67092a-61b5-44f5-b57a-6df58148de2e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.621099 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.621081    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc67092a-61b5-44f5-b57a-6df58148de2e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.626737 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.626714    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb49s\" (UniqueName: \"kubernetes.io/projected/bc67092a-61b5-44f5-b57a-6df58148de2e-kube-api-access-wb49s\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-42t64\" (UID: \"bc67092a-61b5-44f5-b57a-6df58148de2e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.731035 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.730955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:37.862252 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:37.862210    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"]
Jul 28 00:57:37.865058 ip-10-0-140-154 kubenswrapper[2576]: W0728 00:57:37.865031    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc67092a_61b5_44f5_b57a_6df58148de2e.slice/crio-944f831a46022e214e7ec430e536c45281b76d4997937ba19a3c2e878305c5ff WatchSource:0}: Error finding container 944f831a46022e214e7ec430e536c45281b76d4997937ba19a3c2e878305c5ff: Status 404 returned error can't find the container with id 944f831a46022e214e7ec430e536c45281b76d4997937ba19a3c2e878305c5ff
Jul 28 00:57:38.636579 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:38.636544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64" event={"ID":"bc67092a-61b5-44f5-b57a-6df58148de2e","Type":"ContainerStarted","Data":"aaa41670f113c59555c608a923c1733d9ed0a5e1c80f4421b050aebebaa7acb2"}
Jul 28 00:57:38.636966 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:38.636591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64" event={"ID":"bc67092a-61b5-44f5-b57a-6df58148de2e","Type":"ContainerStarted","Data":"944f831a46022e214e7ec430e536c45281b76d4997937ba19a3c2e878305c5ff"}
Jul 28 00:57:38.655293 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:38.655245    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64" podStartSLOduration=1.482213075 podStartE2EDuration="1.655211947s" totalImagesPullingTime="172.998872ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 00:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 00:57:38.654314635 +0000 UTC m=+663.403453359" watchObservedRunningTime="2026-07-28 00:57:38.655211947 +0000 UTC m=+663.404350671"
Jul 28 00:57:40.030550 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:40.030515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:40.030949 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:40.030580    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:40.043695 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:40.043664    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:40.657297 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:40.657267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-wmbxr"
Jul 28 00:57:47.731216 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:47.731179    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:47.731606 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:47.731256    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:47.744014 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:47.743989    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:57:48.692491 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:57:48.692461    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-42t64"
Jul 28 00:58:16.892517 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:16.892473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:58:38.707983 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:38.707945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:58:38.807414 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:38.807382    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:58:39.603863 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.603829    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-798bf9d948-xzpxj"]
Jul 28 00:58:39.604055 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.604013    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-798bf9d948-xzpxj" podUID="526164d7-acab-4bc5-8f82-eb6ffad2b692" containerName="authorino" containerID="cri-o://8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7" gracePeriod=30
Jul 28 00:58:39.792859 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.792820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:58:39.848654 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.848630    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:58:39.894704 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.894629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:58:39.904380 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.904355    2576 generic.go:356] "Generic (PLEG): container finished" podID="526164d7-acab-4bc5-8f82-eb6ffad2b692" containerID="8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7" exitCode=0
Jul 28 00:58:39.904514 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.904388    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798bf9d948-xzpxj"
Jul 28 00:58:39.904514 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.904437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798bf9d948-xzpxj" event={"ID":"526164d7-acab-4bc5-8f82-eb6ffad2b692","Type":"ContainerDied","Data":"8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7"}
Jul 28 00:58:39.904514 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.904476    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798bf9d948-xzpxj" event={"ID":"526164d7-acab-4bc5-8f82-eb6ffad2b692","Type":"ContainerDied","Data":"1d21b73d103fa0b0275c699380fe2007ebda2065a33c169738d24b2a0666c546"}
Jul 28 00:58:39.904514 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.904494    2576 scope.go:117] "RemoveContainer" containerID="8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7"
Jul 28 00:58:39.915549 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.915530    2576 scope.go:117] "RemoveContainer" containerID="8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7"
Jul 28 00:58:39.915821 ip-10-0-140-154 kubenswrapper[2576]: E0728 00:58:39.915791    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7\": container with ID starting with 8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7 not found: ID does not exist" containerID="8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7"
Jul 28 00:58:39.915890 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.915818    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7"} err="failed to get container status \"8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7\": rpc error: code = NotFound desc = could not find container \"8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7\": container with ID starting with 8061ebf7104ae62faec0a2e4f33d213aae332750db7db162387e01efdd5c87a7 not found: ID does not exist"
Jul 28 00:58:39.918197 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.918175    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvphq\" (UniqueName: \"kubernetes.io/projected/526164d7-acab-4bc5-8f82-eb6ffad2b692-kube-api-access-jvphq\") pod \"526164d7-acab-4bc5-8f82-eb6ffad2b692\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") "
Jul 28 00:58:39.918313 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.918297    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/526164d7-acab-4bc5-8f82-eb6ffad2b692-tls-cert\") pod \"526164d7-acab-4bc5-8f82-eb6ffad2b692\" (UID: \"526164d7-acab-4bc5-8f82-eb6ffad2b692\") "
Jul 28 00:58:39.920047 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.920016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/526164d7-acab-4bc5-8f82-eb6ffad2b692-kube-api-access-jvphq" (OuterVolumeSpecName: "kube-api-access-jvphq") pod "526164d7-acab-4bc5-8f82-eb6ffad2b692" (UID: "526164d7-acab-4bc5-8f82-eb6ffad2b692"). InnerVolumeSpecName "kube-api-access-jvphq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 00:58:39.927193 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:39.927165    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/526164d7-acab-4bc5-8f82-eb6ffad2b692-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "526164d7-acab-4bc5-8f82-eb6ffad2b692" (UID: "526164d7-acab-4bc5-8f82-eb6ffad2b692"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 00:58:40.019375 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:40.019356    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvphq\" (UniqueName: \"kubernetes.io/projected/526164d7-acab-4bc5-8f82-eb6ffad2b692-kube-api-access-jvphq\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:58:40.019375 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:40.019376    2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/526164d7-acab-4bc5-8f82-eb6ffad2b692-tls-cert\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 00:58:40.228100 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:40.228072    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-798bf9d948-xzpxj"]
Jul 28 00:58:40.231118 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:40.231100    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-798bf9d948-xzpxj"]
Jul 28 00:58:41.750642 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:41.750608    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="526164d7-acab-4bc5-8f82-eb6ffad2b692" path="/var/lib/kubelet/pods/526164d7-acab-4bc5-8f82-eb6ffad2b692/volumes"
Jul 28 00:58:57.673480 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:57.673442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:58:57.723185 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:58:57.723156    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 00:59:22.391335 ip-10-0-140-154 kubenswrapper[2576]: I0728 00:59:22.391301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:00:04.678283 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:00:04.678185    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:00:04.729129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:00:04.729099    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:00:43.725243 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:00:43.725188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:00:44.940804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:00:44.940776    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:19.155901 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:19.155869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:19.255674 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:19.255640    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:19.375864 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:19.375832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:25.166326 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:25.166286    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:25.426041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:25.425963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:25.528773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:25.528739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:25.637567 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:25.637534    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:31.603213 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:31.603179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:01:31.702137 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:01:31.702100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:02:15.950938 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:02:15.950898    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:05:11.129530 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:11.129497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:05:11.182525 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:11.182489    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:05:35.614656 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:35.614601    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"]
Jul 28 01:05:35.615241 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:35.614851    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v" podUID="f0ab07e3-97db-46b5-be1f-169e5ab12e42" containerName="manager" containerID="cri-o://2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0" gracePeriod=10
Jul 28 01:05:38.459623 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.459599    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 01:05:38.591720 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.591691    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ab07e3-97db-46b5-be1f-169e5ab12e42-extensions-socket-volume\") pod \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") "
Jul 28 01:05:38.591898 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.591753    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6s4jd\" (UniqueName: \"kubernetes.io/projected/f0ab07e3-97db-46b5-be1f-169e5ab12e42-kube-api-access-6s4jd\") pod \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\" (UID: \"f0ab07e3-97db-46b5-be1f-169e5ab12e42\") "
Jul 28 01:05:38.592101 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.592080    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0ab07e3-97db-46b5-be1f-169e5ab12e42-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0ab07e3-97db-46b5-be1f-169e5ab12e42" (UID: "f0ab07e3-97db-46b5-be1f-169e5ab12e42"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:05:38.593622 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.593595    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0ab07e3-97db-46b5-be1f-169e5ab12e42-kube-api-access-6s4jd" (OuterVolumeSpecName: "kube-api-access-6s4jd") pod "f0ab07e3-97db-46b5-be1f-169e5ab12e42" (UID: "f0ab07e3-97db-46b5-be1f-169e5ab12e42"). InnerVolumeSpecName "kube-api-access-6s4jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:05:38.693303 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.693276    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0ab07e3-97db-46b5-be1f-169e5ab12e42-extensions-socket-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:05:38.693303 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.693298    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s4jd\" (UniqueName: \"kubernetes.io/projected/f0ab07e3-97db-46b5-be1f-169e5ab12e42-kube-api-access-6s4jd\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:05:38.714050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.714021    2576 generic.go:356] "Generic (PLEG): container finished" podID="f0ab07e3-97db-46b5-be1f-169e5ab12e42" containerID="2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0" exitCode=0
Jul 28 01:05:38.714156 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.714073    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v" event={"ID":"f0ab07e3-97db-46b5-be1f-169e5ab12e42","Type":"ContainerDied","Data":"2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0"}
Jul 28 01:05:38.714156 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.714110    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v" event={"ID":"f0ab07e3-97db-46b5-be1f-169e5ab12e42","Type":"ContainerDied","Data":"4f44ffa2cd3ced8c111b4afa1c05b5ce2f085a8bfa71604c14f68da439a48f1e"}
Jul 28 01:05:38.714156 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.714132    2576 scope.go:117] "RemoveContainer" containerID="2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0"
Jul 28 01:05:38.714156 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.714074    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"
Jul 28 01:05:38.724464 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.724448    2576 scope.go:117] "RemoveContainer" containerID="2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0"
Jul 28 01:05:38.724697 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:05:38.724680    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0\": container with ID starting with 2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0 not found: ID does not exist" containerID="2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0"
Jul 28 01:05:38.724765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.724705    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0"} err="failed to get container status \"2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0\": rpc error: code = NotFound desc = could not find container \"2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0\": container with ID starting with 2c96f5dd59f4bc5b237ac87b8e528f23fab58dbd709e8cdf6f6a9705cd6669e0 not found: ID does not exist"
Jul 28 01:05:38.735983 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.735964    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"]
Jul 28 01:05:38.741144 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:38.741124    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-qvh8v"]
Jul 28 01:05:39.749434 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:05:39.749406    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0ab07e3-97db-46b5-be1f-169e5ab12e42" path="/var/lib/kubelet/pods/f0ab07e3-97db-46b5-be1f-169e5ab12e42/volumes"
Jul 28 01:06:36.723363 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723329    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"]
Jul 28 01:06:36.723765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723676    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0ab07e3-97db-46b5-be1f-169e5ab12e42" containerName="manager"
Jul 28 01:06:36.723765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723687    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0ab07e3-97db-46b5-be1f-169e5ab12e42" containerName="manager"
Jul 28 01:06:36.723765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723697    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="526164d7-acab-4bc5-8f82-eb6ffad2b692" containerName="authorino"
Jul 28 01:06:36.723765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723702    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="526164d7-acab-4bc5-8f82-eb6ffad2b692" containerName="authorino"
Jul 28 01:06:36.723765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723751    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0ab07e3-97db-46b5-be1f-169e5ab12e42" containerName="manager"
Jul 28 01:06:36.723765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.723762    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="526164d7-acab-4bc5-8f82-eb6ffad2b692" containerName="authorino"
Jul 28 01:06:36.726907 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.726889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:36.729634 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.729604    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 01:06:36.729746 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.729680    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 01:06:36.729746 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.729706    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-b5q2g\""
Jul 28 01:06:36.737422 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.737398    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"]
Jul 28 01:06:36.754119 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.754094    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f149aef-5e7c-41ea-9860-14a8863a1adf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k7jm2\" (UID: \"3f149aef-5e7c-41ea-9860-14a8863a1adf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:36.754260 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.754177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndclt\" (UniqueName: \"kubernetes.io/projected/3f149aef-5e7c-41ea-9860-14a8863a1adf-kube-api-access-ndclt\") pod \"kuadrant-operator-controller-manager-86878c769f-k7jm2\" (UID: \"3f149aef-5e7c-41ea-9860-14a8863a1adf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:36.854844 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.854816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndclt\" (UniqueName: \"kubernetes.io/projected/3f149aef-5e7c-41ea-9860-14a8863a1adf-kube-api-access-ndclt\") pod \"kuadrant-operator-controller-manager-86878c769f-k7jm2\" (UID: \"3f149aef-5e7c-41ea-9860-14a8863a1adf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:36.854977 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.854900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f149aef-5e7c-41ea-9860-14a8863a1adf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k7jm2\" (UID: \"3f149aef-5e7c-41ea-9860-14a8863a1adf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:36.855256 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.855236    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3f149aef-5e7c-41ea-9860-14a8863a1adf-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-k7jm2\" (UID: \"3f149aef-5e7c-41ea-9860-14a8863a1adf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:36.862670 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:36.862644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndclt\" (UniqueName: \"kubernetes.io/projected/3f149aef-5e7c-41ea-9860-14a8863a1adf-kube-api-access-ndclt\") pod \"kuadrant-operator-controller-manager-86878c769f-k7jm2\" (UID: \"3f149aef-5e7c-41ea-9860-14a8863a1adf\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:37.037867 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:37.037789    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:37.188801 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:37.188771    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"]
Jul 28 01:06:37.191399 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:06:37.191368    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f149aef_5e7c_41ea_9860_14a8863a1adf.slice/crio-8ec3d15529cc7d46cdddef1579b4dbb085094620dd7b34b5fffc014a03a55853 WatchSource:0}: Error finding container 8ec3d15529cc7d46cdddef1579b4dbb085094620dd7b34b5fffc014a03a55853: Status 404 returned error can't find the container with id 8ec3d15529cc7d46cdddef1579b4dbb085094620dd7b34b5fffc014a03a55853
Jul 28 01:06:37.968546 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:37.968507    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2" event={"ID":"3f149aef-5e7c-41ea-9860-14a8863a1adf","Type":"ContainerStarted","Data":"0b3c5015a1175b5d8bb73ec15d689847559358cd4ca2200dc8c881b732372165"}
Jul 28 01:06:37.968998 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:37.968620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2" event={"ID":"3f149aef-5e7c-41ea-9860-14a8863a1adf","Type":"ContainerStarted","Data":"8ec3d15529cc7d46cdddef1579b4dbb085094620dd7b34b5fffc014a03a55853"}
Jul 28 01:06:37.968998 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:37.968673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:06:37.991925 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:37.991879    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2" podStartSLOduration=1.991865963 podStartE2EDuration="1.991865963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:06:37.98942291 +0000 UTC m=+1202.738561637" watchObservedRunningTime="2026-07-28 01:06:37.991865963 +0000 UTC m=+1202.741004687"
Jul 28 01:06:48.975058 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:06:48.975029    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-k7jm2"
Jul 28 01:08:04.618510 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:08:04.618474    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:08:43.701104 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:08:43.701065    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:09:20.931784 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:20.931712    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"]
Jul 28 01:09:20.934811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:20.934794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:20.937191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:20.937173    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-fa8a1ecc-kserve-self-signed-certs\""
Jul 28 01:09:20.950279 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:20.950258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"]
Jul 28 01:09:21.027115 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.027090    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-dshm\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.027269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.027125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tls-certs\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.027269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.027147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26vrx\" (UniqueName: \"kubernetes.io/projected/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-kube-api-access-26vrx\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.027269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.027169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-home\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.027269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.027251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tmp-dir\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.027404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.027296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-model-cache\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.096759 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.096727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:09:21.128562 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.128534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-dshm\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.128643 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.128573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tls-certs\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.128643 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.128593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26vrx\" (UniqueName: \"kubernetes.io/projected/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-kube-api-access-26vrx\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.128643 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.128622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-home\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.128750 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.128722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tmp-dir\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.128794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.128781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-model-cache\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.129060 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.129040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-home\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.129144 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.129094    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-model-cache\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.129195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.129138    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tmp-dir\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.130801 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.130783    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-dshm\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.131052 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.131035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tls-certs\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.145585 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.145560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26vrx\" (UniqueName: \"kubernetes.io/projected/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-kube-api-access-26vrx\") pod \"e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") " pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.243831 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.243760    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:21.374499 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.374472    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"]
Jul 28 01:09:21.376407 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:09:21.376380    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76f1ff20_5afd_4ed3_9d6d_b69cfc54048d.slice/crio-aa885cff239b5748578f9c71d9fe92240794463a4b9a35251962daf8f15c9e37 WatchSource:0}: Error finding container aa885cff239b5748578f9c71d9fe92240794463a4b9a35251962daf8f15c9e37: Status 404 returned error can't find the container with id aa885cff239b5748578f9c71d9fe92240794463a4b9a35251962daf8f15c9e37
Jul 28 01:09:21.676821 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.676786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5" event={"ID":"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d","Type":"ContainerStarted","Data":"ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e"}
Jul 28 01:09:21.677006 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.676831    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5" event={"ID":"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d","Type":"ContainerStarted","Data":"aa885cff239b5748578f9c71d9fe92240794463a4b9a35251962daf8f15c9e37"}
Jul 28 01:09:21.695443 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:21.695399    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5" podStartSLOduration=1.695386195 podStartE2EDuration="1.695386195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:09:21.693516613 +0000 UTC m=+1366.442655338" watchObservedRunningTime="2026-07-28 01:09:21.695386195 +0000 UTC m=+1366.444524919"
Jul 28 01:09:27.535699 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:27.535668    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:09:31.244378 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:31.244341    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:31.244860 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:31.244403    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:31.256864 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:31.256834    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:09:31.739829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:09:31.739802    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:10:00.100650 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:00.100612    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"]
Jul 28 01:10:00.124294 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:00.124264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:10:01.216293 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:01.216262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:10:15.120653 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.120611    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5" podUID="76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" containerName="main" containerID="cri-o://ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e" gracePeriod=45
Jul 28 01:10:15.370137 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.370112    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:10:15.480113 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480011    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tmp-dir\") pod \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") "
Jul 28 01:10:15.480113 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480070    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tls-certs\") pod \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") "
Jul 28 01:10:15.480113 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480097    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-dshm\") pod \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") "
Jul 28 01:10:15.480868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480120    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-home\") pod \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") "
Jul 28 01:10:15.480868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480179    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-model-cache\") pod \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") "
Jul 28 01:10:15.480868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26vrx\" (UniqueName: \"kubernetes.io/projected/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-kube-api-access-26vrx\") pod \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\" (UID: \"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d\") "
Jul 28 01:10:15.480868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480503    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-home" (OuterVolumeSpecName: "home") pod "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" (UID: "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:10:15.480868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480532    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-model-cache" (OuterVolumeSpecName: "model-cache") pod "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" (UID: "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:10:15.480868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.480591    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" (UID: "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:10:15.482504 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.482467    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-kube-api-access-26vrx" (OuterVolumeSpecName: "kube-api-access-26vrx") pod "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" (UID: "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d"). InnerVolumeSpecName "kube-api-access-26vrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:10:15.482504 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.482489    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" (UID: "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:10:15.482636 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.482493    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-dshm" (OuterVolumeSpecName: "dshm") pod "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" (UID: "76f1ff20-5afd-4ed3-9d6d-b69cfc54048d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:10:15.587361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.582258    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:10:15.587361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.582296    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:10:15.587361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.582309    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:10:15.587361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.582321    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:10:15.587361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.582333    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26vrx\" (UniqueName: \"kubernetes.io/projected/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-kube-api-access-26vrx\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:10:15.587361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.582352    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:10:15.915954 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.915916    2576 generic.go:356] "Generic (PLEG): container finished" podID="76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" containerID="ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e" exitCode=0
Jul 28 01:10:15.916144 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.915972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5" event={"ID":"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d","Type":"ContainerDied","Data":"ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e"}
Jul 28 01:10:15.916144 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.916007    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5" event={"ID":"76f1ff20-5afd-4ed3-9d6d-b69cfc54048d","Type":"ContainerDied","Data":"aa885cff239b5748578f9c71d9fe92240794463a4b9a35251962daf8f15c9e37"}
Jul 28 01:10:15.916144 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.916028    2576 scope.go:117] "RemoveContainer" containerID="ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e"
Jul 28 01:10:15.916144 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.915973    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"
Jul 28 01:10:15.924445 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.924425    2576 scope.go:117] "RemoveContainer" containerID="ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e"
Jul 28 01:10:15.924719 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:10:15.924700    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e\": container with ID starting with ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e not found: ID does not exist" containerID="ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e"
Jul 28 01:10:15.924781 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.924733    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e"} err="failed to get container status \"ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e\": rpc error: code = NotFound desc = could not find container \"ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e\": container with ID starting with ea6d39f1635b44db302ad2434036cc9bd4ec929a9fe2003e0e6a05068dd4333e not found: ID does not exist"
Jul 28 01:10:15.931868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.931813    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"]
Jul 28 01:10:15.934025 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:15.933999    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-fa8a1ecc-kserve-5494f9b6ff-fkgk5"]
Jul 28 01:10:17.750795 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:10:17.750761    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" path="/var/lib/kubelet/pods/76f1ff20-5afd-4ed3-9d6d-b69cfc54048d/volumes"
Jul 28 01:11:41.170989 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:11:41.170952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:05.809069 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:05.809030    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:42.943197 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:42.943160    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:45.896450 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:45.896365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:45.984416 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:45.984381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:46.323040 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:46.323012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:52.188348 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:52.188312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:52.253026 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:52.253001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:52.425611 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:52.425584    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:52.699592 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:52.699563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:52.837258 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:52.837214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:55.302234 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:55.302189    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:55.406277 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:55.406243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:55.665804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:55.665772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:56.059909 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.059864    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:56.236077 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.236047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:56.327137 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.327070    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"]
Jul 28 01:13:56.327521 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.327465    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" containerName="main"
Jul 28 01:13:56.327521 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.327482    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" containerName="main"
Jul 28 01:13:56.327604 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.327588    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f1ff20-5afd-4ed3-9d6d-b69cfc54048d" containerName="main"
Jul 28 01:13:56.328441 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.328420    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:13:56.330933 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.330917    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.333518 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.333501    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1dcfab82-dockercfg-zgsgj\""
Jul 28 01:13:56.333678 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.333663    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1dcfab82\""
Jul 28 01:13:56.344576 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.344556    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"]
Jul 28 01:13:56.366470 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.366446    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"]
Jul 28 01:13:56.369745 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.369730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.383763 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.383737    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"]
Jul 28 01:13:56.415564 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.415541    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.415665 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.415590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v64ds\" (UniqueName: \"kubernetes.io/projected/9f206013-133d-438c-8d0b-5a80670d96dd-kube-api-access-v64ds\") pod \"payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.415665 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.415654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f206013-133d-438c-8d0b-5a80670d96dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.415766 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.415683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx25v\" (UniqueName: \"kubernetes.io/projected/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-kube-api-access-xx25v\") pod \"payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.516068 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.516043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f206013-133d-438c-8d0b-5a80670d96dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.516186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.516081    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx25v\" (UniqueName: \"kubernetes.io/projected/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-kube-api-access-xx25v\") pod \"payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.516186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.516110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.516186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.516149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v64ds\" (UniqueName: \"kubernetes.io/projected/9f206013-133d-438c-8d0b-5a80670d96dd-kube-api-access-v64ds\") pod \"payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.516855 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.516833    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f206013-133d-438c-8d0b-5a80670d96dd-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.516931 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.516840    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-plugins-config-volume\") pod \"payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.523904 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.523883    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v64ds\" (UniqueName: \"kubernetes.io/projected/9f206013-133d-438c-8d0b-5a80670d96dd-kube-api-access-v64ds\") pod \"payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.523986 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.523969    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx25v\" (UniqueName: \"kubernetes.io/projected/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-kube-api-access-xx25v\") pod \"payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") " pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.641838 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.641771    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.678853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.678821    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:56.786534 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.786502    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"]
Jul 28 01:13:56.787789 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:13:56.787753    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f206013_133d_438c_8d0b_5a80670d96dd.slice/crio-396a74e568a33685f750268cf5a9cd42cd9ccd83e30f981590fb593294375d49 WatchSource:0}: Error finding container 396a74e568a33685f750268cf5a9cd42cd9ccd83e30f981590fb593294375d49: Status 404 returned error can't find the container with id 396a74e568a33685f750268cf5a9cd42cd9ccd83e30f981590fb593294375d49
Jul 28 01:13:56.873861 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.873836    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m" event={"ID":"9f206013-133d-438c-8d0b-5a80670d96dd","Type":"ContainerStarted","Data":"ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb"}
Jul 28 01:13:56.873971 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.873881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m" event={"ID":"9f206013-133d-438c-8d0b-5a80670d96dd","Type":"ContainerStarted","Data":"396a74e568a33685f750268cf5a9cd42cd9ccd83e30f981590fb593294375d49"}
Jul 28 01:13:56.873971 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.873956    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:13:56.895839 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:56.895765    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m" podStartSLOduration=0.895750693 podStartE2EDuration="895.750693ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:13:56.894390578 +0000 UTC m=+1641.643529301" watchObservedRunningTime="2026-07-28 01:13:56.895750693 +0000 UTC m=+1641.644889417"
Jul 28 01:13:57.050499 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:57.050356    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"]
Jul 28 01:13:57.052688 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:13:57.052658    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fd4580_62a1_4e21_b4bd_8c011c67ea02.slice/crio-85e5a323362f0f24d3e3b4a2ca2d565fdade3723f05ca6902d29deb30eaecbbd WatchSource:0}: Error finding container 85e5a323362f0f24d3e3b4a2ca2d565fdade3723f05ca6902d29deb30eaecbbd: Status 404 returned error can't find the container with id 85e5a323362f0f24d3e3b4a2ca2d565fdade3723f05ca6902d29deb30eaecbbd
Jul 28 01:13:57.880817 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:57.880779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc" event={"ID":"d4fd4580-62a1-4e21-b4bd-8c011c67ea02","Type":"ContainerStarted","Data":"cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e"}
Jul 28 01:13:57.881290 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:57.880931    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:13:57.881290 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:57.880956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc" event={"ID":"d4fd4580-62a1-4e21-b4bd-8c011c67ea02","Type":"ContainerStarted","Data":"85e5a323362f0f24d3e3b4a2ca2d565fdade3723f05ca6902d29deb30eaecbbd"}
Jul 28 01:13:57.898426 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:13:57.898377    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc" podStartSLOduration=1.898363212 podStartE2EDuration="1.898363212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:13:57.896447049 +0000 UTC m=+1642.645585790" watchObservedRunningTime="2026-07-28 01:13:57.898363212 +0000 UTC m=+1642.647501936"
Jul 28 01:14:07.882864 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:07.882834    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:14:08.887327 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:08.887299    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:14:22.335596 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:22.335561    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:22.387194 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:22.387162    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:31.330782 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.330742    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:31.352349 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.352318    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"]
Jul 28 01:14:31.352652 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.352623    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc" podUID="d4fd4580-62a1-4e21-b4bd-8c011c67ea02" containerName="payload-processing" containerID="cri-o://cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e" gracePeriod=30
Jul 28 01:14:31.358861 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.358832    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"]
Jul 28 01:14:31.359117 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.359072    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m" podUID="9f206013-133d-438c-8d0b-5a80670d96dd" containerName="payload-pre-processing" containerID="cri-o://ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb" gracePeriod=30
Jul 28 01:14:31.431071 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.431040    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:31.486784 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.486756    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:31.632784 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.632762    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:14:31.635525 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.635507    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:14:31.710302 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.710276    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xx25v\" (UniqueName: \"kubernetes.io/projected/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-kube-api-access-xx25v\") pod \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") "
Jul 28 01:14:31.710465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.710328    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-plugins-config-volume\") pod \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\" (UID: \"d4fd4580-62a1-4e21-b4bd-8c011c67ea02\") "
Jul 28 01:14:31.710465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.710371    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f206013-133d-438c-8d0b-5a80670d96dd-plugins-config-volume\") pod \"9f206013-133d-438c-8d0b-5a80670d96dd\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") "
Jul 28 01:14:31.710465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.710388    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v64ds\" (UniqueName: \"kubernetes.io/projected/9f206013-133d-438c-8d0b-5a80670d96dd-kube-api-access-v64ds\") pod \"9f206013-133d-438c-8d0b-5a80670d96dd\" (UID: \"9f206013-133d-438c-8d0b-5a80670d96dd\") "
Jul 28 01:14:31.710736 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.710716    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d4fd4580-62a1-4e21-b4bd-8c011c67ea02" (UID: "d4fd4580-62a1-4e21-b4bd-8c011c67ea02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:14:31.710808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.710768    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f206013-133d-438c-8d0b-5a80670d96dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f206013-133d-438c-8d0b-5a80670d96dd" (UID: "9f206013-133d-438c-8d0b-5a80670d96dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:14:31.712514 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.712497    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-kube-api-access-xx25v" (OuterVolumeSpecName: "kube-api-access-xx25v") pod "d4fd4580-62a1-4e21-b4bd-8c011c67ea02" (UID: "d4fd4580-62a1-4e21-b4bd-8c011c67ea02"). InnerVolumeSpecName "kube-api-access-xx25v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:14:31.712590 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.712546    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f206013-133d-438c-8d0b-5a80670d96dd-kube-api-access-v64ds" (OuterVolumeSpecName: "kube-api-access-v64ds") pod "9f206013-133d-438c-8d0b-5a80670d96dd" (UID: "9f206013-133d-438c-8d0b-5a80670d96dd"). InnerVolumeSpecName "kube-api-access-v64ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:14:31.810966 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.810935    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx25v\" (UniqueName: \"kubernetes.io/projected/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-kube-api-access-xx25v\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:14:31.810966 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.810961    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4fd4580-62a1-4e21-b4bd-8c011c67ea02-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:14:31.811120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.810974    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f206013-133d-438c-8d0b-5a80670d96dd-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:14:31.811120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:31.810986    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v64ds\" (UniqueName: \"kubernetes.io/projected/9f206013-133d-438c-8d0b-5a80670d96dd-kube-api-access-v64ds\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:14:32.028695 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.028612    2576 generic.go:356] "Generic (PLEG): container finished" podID="d4fd4580-62a1-4e21-b4bd-8c011c67ea02" containerID="cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e" exitCode=0
Jul 28 01:14:32.028695 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.028651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"
Jul 28 01:14:32.028908 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.028701    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc" event={"ID":"d4fd4580-62a1-4e21-b4bd-8c011c67ea02","Type":"ContainerDied","Data":"cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e"}
Jul 28 01:14:32.028908 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.028752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc" event={"ID":"d4fd4580-62a1-4e21-b4bd-8c011c67ea02","Type":"ContainerDied","Data":"85e5a323362f0f24d3e3b4a2ca2d565fdade3723f05ca6902d29deb30eaecbbd"}
Jul 28 01:14:32.028908 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.028779    2576 scope.go:117] "RemoveContainer" containerID="cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e"
Jul 28 01:14:32.030070 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.030049    2576 generic.go:356] "Generic (PLEG): container finished" podID="9f206013-133d-438c-8d0b-5a80670d96dd" containerID="ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb" exitCode=0
Jul 28 01:14:32.030171 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.030091    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m" event={"ID":"9f206013-133d-438c-8d0b-5a80670d96dd","Type":"ContainerDied","Data":"ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb"}
Jul 28 01:14:32.030171 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.030098    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"
Jul 28 01:14:32.030171 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.030125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m" event={"ID":"9f206013-133d-438c-8d0b-5a80670d96dd","Type":"ContainerDied","Data":"396a74e568a33685f750268cf5a9cd42cd9ccd83e30f981590fb593294375d49"}
Jul 28 01:14:32.037162 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.037146    2576 scope.go:117] "RemoveContainer" containerID="cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e"
Jul 28 01:14:32.037464 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:14:32.037425    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e\": container with ID starting with cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e not found: ID does not exist" containerID="cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e"
Jul 28 01:14:32.037565 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.037484    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e"} err="failed to get container status \"cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e\": rpc error: code = NotFound desc = could not find container \"cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e\": container with ID starting with cc7e5164dedd2f93a654c9158ba7ec5b2c8fb9a768fd6004d1398fb0b8aaca2e not found: ID does not exist"
Jul 28 01:14:32.037565 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.037507    2576 scope.go:117] "RemoveContainer" containerID="ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb"
Jul 28 01:14:32.044836 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.044818    2576 scope.go:117] "RemoveContainer" containerID="ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb"
Jul 28 01:14:32.045773 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:14:32.045705    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb\": container with ID starting with ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb not found: ID does not exist" containerID="ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb"
Jul 28 01:14:32.045773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.045735    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb"} err="failed to get container status \"ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb\": rpc error: code = NotFound desc = could not find container \"ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb\": container with ID starting with ef9d837177098e0d21474dc66858e1277bc42352ff1d2733394baf2208a45bcb not found: ID does not exist"
Jul 28 01:14:32.047420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.047401    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"]
Jul 28 01:14:32.048929 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.048908    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1dcfab82-779b56886d-t284m"]
Jul 28 01:14:32.058348 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.058326    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"]
Jul 28 01:14:32.061566 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:32.061545    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1dcfab82-646f78f6f4-ppvnc"]
Jul 28 01:14:33.750970 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:33.750934    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f206013-133d-438c-8d0b-5a80670d96dd" path="/var/lib/kubelet/pods/9f206013-133d-438c-8d0b-5a80670d96dd/volumes"
Jul 28 01:14:33.751474 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:33.751271    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4fd4580-62a1-4e21-b4bd-8c011c67ea02" path="/var/lib/kubelet/pods/d4fd4580-62a1-4e21-b4bd-8c011c67ea02/volumes"
Jul 28 01:14:35.820213 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:35.820186    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:35.970555 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:35.970524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:36.255860 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:36.255833    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:36.442784 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:36.442753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:36.533923 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:36.533848    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:36.656357 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:36.656323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:36.996769 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:36.996737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:37.287838 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:37.287759    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:42.487666 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:42.487632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:42.590631 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:42.590586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:42.811182 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:42.811158    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:43.098736 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.098661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:43.199852 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.199778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:14:43.205801 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.205779    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"]
Jul 28 01:14:43.206138 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.206125    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4fd4580-62a1-4e21-b4bd-8c011c67ea02" containerName="payload-processing"
Jul 28 01:14:43.206181 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.206141    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4fd4580-62a1-4e21-b4bd-8c011c67ea02" containerName="payload-processing"
Jul 28 01:14:43.206181 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.206157    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f206013-133d-438c-8d0b-5a80670d96dd" containerName="payload-pre-processing"
Jul 28 01:14:43.206181 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.206162    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f206013-133d-438c-8d0b-5a80670d96dd" containerName="payload-pre-processing"
Jul 28 01:14:43.206291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.206231    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f206013-133d-438c-8d0b-5a80670d96dd" containerName="payload-pre-processing"
Jul 28 01:14:43.206291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.206243    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4fd4580-62a1-4e21-b4bd-8c011c67ea02" containerName="payload-processing"
Jul 28 01:14:43.209121 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.209104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.211741 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.211717    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d990b63a-dockercfg-7zfrv\""
Jul 28 01:14:43.211839 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.211786    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d990b63a\""
Jul 28 01:14:43.221808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.221787    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"]
Jul 28 01:14:43.225065 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.225045    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.226269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.226251    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"]
Jul 28 01:14:43.236333 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.236314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"]
Jul 28 01:14:43.309690 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.309663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2chkh\" (UniqueName: \"kubernetes.io/projected/9f44d3b7-a65e-4b8e-a699-3ce74344960e-kube-api-access-2chkh\") pod \"payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.309809 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.309738    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p75s\" (UniqueName: \"kubernetes.io/projected/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-kube-api-access-5p75s\") pod \"payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") " pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.309851 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.309813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") " pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.309851 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.309842    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44d3b7-a65e-4b8e-a699-3ce74344960e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.411255 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.411179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p75s\" (UniqueName: \"kubernetes.io/projected/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-kube-api-access-5p75s\") pod \"payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") " pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.411255 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.411217    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") " pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.411421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.411269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44d3b7-a65e-4b8e-a699-3ce74344960e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.411421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.411319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2chkh\" (UniqueName: \"kubernetes.io/projected/9f44d3b7-a65e-4b8e-a699-3ce74344960e-kube-api-access-2chkh\") pod \"payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.411914 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.411898    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-plugins-config-volume\") pod \"payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") " pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.411983 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.411967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44d3b7-a65e-4b8e-a699-3ce74344960e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.422873 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.422849    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p75s\" (UniqueName: \"kubernetes.io/projected/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-kube-api-access-5p75s\") pod \"payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") " pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.424675 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.424655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2chkh\" (UniqueName: \"kubernetes.io/projected/9f44d3b7-a65e-4b8e-a699-3ce74344960e-kube-api-access-2chkh\") pod \"payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.519305 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.519286    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:43.534380 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.534358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:43.681041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.680982    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"]
Jul 28 01:14:43.685565 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:14:43.685531    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d50d324_e8e6_4c5a_91e7_cb2213c4a774.slice/crio-2268a5d781d0729196d4bd50adf05a06624fd5ac6d28f5016d4bc0477591675f WatchSource:0}: Error finding container 2268a5d781d0729196d4bd50adf05a06624fd5ac6d28f5016d4bc0477591675f: Status 404 returned error can't find the container with id 2268a5d781d0729196d4bd50adf05a06624fd5ac6d28f5016d4bc0477591675f
Jul 28 01:14:43.863355 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:43.863327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"]
Jul 28 01:14:43.865693 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:14:43.865667    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f44d3b7_a65e_4b8e_a699_3ce74344960e.slice/crio-7b631a54f94d54cf0d4c7c31d8dee688bf80c0b88a9b0c1bea34a7469b1001c2 WatchSource:0}: Error finding container 7b631a54f94d54cf0d4c7c31d8dee688bf80c0b88a9b0c1bea34a7469b1001c2: Status 404 returned error can't find the container with id 7b631a54f94d54cf0d4c7c31d8dee688bf80c0b88a9b0c1bea34a7469b1001c2
Jul 28 01:14:44.094096 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.094060    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v" event={"ID":"9f44d3b7-a65e-4b8e-a699-3ce74344960e","Type":"ContainerStarted","Data":"fdd7e021844ee8c4e16a82379a2436449414b2ec9a84a4f251e17eceb801ae8c"}
Jul 28 01:14:44.094346 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.094248    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:14:44.094346 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.094293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v" event={"ID":"9f44d3b7-a65e-4b8e-a699-3ce74344960e","Type":"ContainerStarted","Data":"7b631a54f94d54cf0d4c7c31d8dee688bf80c0b88a9b0c1bea34a7469b1001c2"}
Jul 28 01:14:44.095526 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.095500    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb" event={"ID":"3d50d324-e8e6-4c5a-91e7-cb2213c4a774","Type":"ContainerStarted","Data":"ec284aded6d90c5dd1fc99be759620b4a95e96e0ccca44bce43148153ce8b92c"}
Jul 28 01:14:44.095660 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.095542    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb" event={"ID":"3d50d324-e8e6-4c5a-91e7-cb2213c4a774","Type":"ContainerStarted","Data":"2268a5d781d0729196d4bd50adf05a06624fd5ac6d28f5016d4bc0477591675f"}
Jul 28 01:14:44.095660 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.095576    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:44.109848 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.109806    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v" podStartSLOduration=1.109794458 podStartE2EDuration="1.109794458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:14:44.10883466 +0000 UTC m=+1688.857973385" watchObservedRunningTime="2026-07-28 01:14:44.109794458 +0000 UTC m=+1688.858933182"
Jul 28 01:14:44.124013 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:44.123972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb" podStartSLOduration=1.123959437 podStartE2EDuration="1.123959437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:14:44.122970588 +0000 UTC m=+1688.872109311" watchObservedRunningTime="2026-07-28 01:14:44.123959437 +0000 UTC m=+1688.873098161"
Jul 28 01:14:55.101556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:55.101521    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:14:55.102047 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:14:55.101674    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:15:18.001716 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.001680    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:18.027314 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.027286    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"]
Jul 28 01:15:18.027556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.027528    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb" podUID="3d50d324-e8e6-4c5a-91e7-cb2213c4a774" containerName="payload-processing" containerID="cri-o://ec284aded6d90c5dd1fc99be759620b4a95e96e0ccca44bce43148153ce8b92c" gracePeriod=30
Jul 28 01:15:18.035187 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.035033    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"]
Jul 28 01:15:18.035304 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.035258    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v" podUID="9f44d3b7-a65e-4b8e-a699-3ce74344960e" containerName="payload-pre-processing" containerID="cri-o://fdd7e021844ee8c4e16a82379a2436449414b2ec9a84a4f251e17eceb801ae8c" gracePeriod=30
Jul 28 01:15:18.066898 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.066871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:18.122280 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.122237    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:18.246856 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.246828    2576 generic.go:356] "Generic (PLEG): container finished" podID="9f44d3b7-a65e-4b8e-a699-3ce74344960e" containerID="fdd7e021844ee8c4e16a82379a2436449414b2ec9a84a4f251e17eceb801ae8c" exitCode=0
Jul 28 01:15:18.247005 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.246920    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v" event={"ID":"9f44d3b7-a65e-4b8e-a699-3ce74344960e","Type":"ContainerDied","Data":"fdd7e021844ee8c4e16a82379a2436449414b2ec9a84a4f251e17eceb801ae8c"}
Jul 28 01:15:18.248697 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.248645    2576 generic.go:356] "Generic (PLEG): container finished" podID="3d50d324-e8e6-4c5a-91e7-cb2213c4a774" containerID="ec284aded6d90c5dd1fc99be759620b4a95e96e0ccca44bce43148153ce8b92c" exitCode=0
Jul 28 01:15:18.248810 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.248701    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb" event={"ID":"3d50d324-e8e6-4c5a-91e7-cb2213c4a774","Type":"ContainerDied","Data":"ec284aded6d90c5dd1fc99be759620b4a95e96e0ccca44bce43148153ce8b92c"}
Jul 28 01:15:18.299676 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.299654    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:15:18.302273 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.302247    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:15:18.400017 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.399986    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2chkh\" (UniqueName: \"kubernetes.io/projected/9f44d3b7-a65e-4b8e-a699-3ce74344960e-kube-api-access-2chkh\") pod \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") "
Jul 28 01:15:18.400153 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.400104    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44d3b7-a65e-4b8e-a699-3ce74344960e-plugins-config-volume\") pod \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\" (UID: \"9f44d3b7-a65e-4b8e-a699-3ce74344960e\") "
Jul 28 01:15:18.400196 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.400154    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p75s\" (UniqueName: \"kubernetes.io/projected/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-kube-api-access-5p75s\") pod \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") "
Jul 28 01:15:18.400286 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.400271    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-plugins-config-volume\") pod \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\" (UID: \"3d50d324-e8e6-4c5a-91e7-cb2213c4a774\") "
Jul 28 01:15:18.400546 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.400520    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f44d3b7-a65e-4b8e-a699-3ce74344960e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9f44d3b7-a65e-4b8e-a699-3ce74344960e" (UID: "9f44d3b7-a65e-4b8e-a699-3ce74344960e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:15:18.400674 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.400651    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d50d324-e8e6-4c5a-91e7-cb2213c4a774" (UID: "3d50d324-e8e6-4c5a-91e7-cb2213c4a774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:15:18.402009 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.401986    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f44d3b7-a65e-4b8e-a699-3ce74344960e-kube-api-access-2chkh" (OuterVolumeSpecName: "kube-api-access-2chkh") pod "9f44d3b7-a65e-4b8e-a699-3ce74344960e" (UID: "9f44d3b7-a65e-4b8e-a699-3ce74344960e"). InnerVolumeSpecName "kube-api-access-2chkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:15:18.402125 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.402107    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-kube-api-access-5p75s" (OuterVolumeSpecName: "kube-api-access-5p75s") pod "3d50d324-e8e6-4c5a-91e7-cb2213c4a774" (UID: "3d50d324-e8e6-4c5a-91e7-cb2213c4a774"). InnerVolumeSpecName "kube-api-access-5p75s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:15:18.501346 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.501317    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p75s\" (UniqueName: \"kubernetes.io/projected/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-kube-api-access-5p75s\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:15:18.501346 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.501341    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d50d324-e8e6-4c5a-91e7-cb2213c4a774-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:15:18.501346 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.501351    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2chkh\" (UniqueName: \"kubernetes.io/projected/9f44d3b7-a65e-4b8e-a699-3ce74344960e-kube-api-access-2chkh\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:15:18.501541 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:18.501360    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f44d3b7-a65e-4b8e-a699-3ce74344960e-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:15:19.256039 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.256010    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"
Jul 28 01:15:19.256487 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.256013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v" event={"ID":"9f44d3b7-a65e-4b8e-a699-3ce74344960e","Type":"ContainerDied","Data":"7b631a54f94d54cf0d4c7c31d8dee688bf80c0b88a9b0c1bea34a7469b1001c2"}
Jul 28 01:15:19.256487 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.256145    2576 scope.go:117] "RemoveContainer" containerID="fdd7e021844ee8c4e16a82379a2436449414b2ec9a84a4f251e17eceb801ae8c"
Jul 28 01:15:19.257512 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.257379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb" event={"ID":"3d50d324-e8e6-4c5a-91e7-cb2213c4a774","Type":"ContainerDied","Data":"2268a5d781d0729196d4bd50adf05a06624fd5ac6d28f5016d4bc0477591675f"}
Jul 28 01:15:19.257512 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.257430    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"
Jul 28 01:15:19.264325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.264288    2576 scope.go:117] "RemoveContainer" containerID="ec284aded6d90c5dd1fc99be759620b4a95e96e0ccca44bce43148153ce8b92c"
Jul 28 01:15:19.280334 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.280309    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"]
Jul 28 01:15:19.283769 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.283748    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-d990b63a-84c5df597-r4dlb"]
Jul 28 01:15:19.293982 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.293936    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"]
Jul 28 01:15:19.296092 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.296074    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d990b63a-6b79647748-jql2v"]
Jul 28 01:15:19.751288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.751247    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d50d324-e8e6-4c5a-91e7-cb2213c4a774" path="/var/lib/kubelet/pods/3d50d324-e8e6-4c5a-91e7-cb2213c4a774/volumes"
Jul 28 01:15:19.751668 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:19.751648    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f44d3b7-a65e-4b8e-a699-3ce74344960e" path="/var/lib/kubelet/pods/9f44d3b7-a65e-4b8e-a699-3ce74344960e/volumes"
Jul 28 01:15:22.549549 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:22.549516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:22.711423 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:22.711393    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:23.002629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:23.002593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:23.172464 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:23.172432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:23.486321 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:23.486261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:23.637291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:23.637255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:25.750291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:25.750204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:25.862648 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:25.862614    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:26.017953 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.017878    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:26.118577 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.118542    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:26.472998 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.472960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:26.561597 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.561565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:15:26.569738 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.569714    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"]
Jul 28 01:15:26.570037 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.570025    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f44d3b7-a65e-4b8e-a699-3ce74344960e" containerName="payload-pre-processing"
Jul 28 01:15:26.570081 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.570039    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f44d3b7-a65e-4b8e-a699-3ce74344960e" containerName="payload-pre-processing"
Jul 28 01:15:26.570081 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.570063    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d50d324-e8e6-4c5a-91e7-cb2213c4a774" containerName="payload-processing"
Jul 28 01:15:26.570081 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.570069    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d50d324-e8e6-4c5a-91e7-cb2213c4a774" containerName="payload-processing"
Jul 28 01:15:26.570178 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.570120    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f44d3b7-a65e-4b8e-a699-3ce74344960e" containerName="payload-pre-processing"
Jul 28 01:15:26.570178 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.570128    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d50d324-e8e6-4c5a-91e7-cb2213c4a774" containerName="payload-processing"
Jul 28 01:15:26.574950 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.574934    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.578017 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.577996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-60243cce-dockercfg-kzxd6\""
Jul 28 01:15:26.578435 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.578420    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-60243cce\""
Jul 28 01:15:26.584382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.584351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"]
Jul 28 01:15:26.614981 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.614960    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"]
Jul 28 01:15:26.618263 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.618248    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.627210 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.627192    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"]
Jul 28 01:15:26.668539 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.668509    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5251119-9377-45db-a145-5c180ccdea47-plugins-config-volume\") pod \"payload-processing-e2e-ait-60243cce-69f485c788-bl85k\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") " pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.668656 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.668569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.668656 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.668636    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqdpl\" (UniqueName: \"kubernetes.io/projected/c5251119-9377-45db-a145-5c180ccdea47-kube-api-access-lqdpl\") pod \"payload-processing-e2e-ait-60243cce-69f485c788-bl85k\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") " pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.668748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.668669    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbm7m\" (UniqueName: \"kubernetes.io/projected/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-kube-api-access-qbm7m\") pod \"payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.769935 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.769870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5251119-9377-45db-a145-5c180ccdea47-plugins-config-volume\") pod \"payload-processing-e2e-ait-60243cce-69f485c788-bl85k\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") " pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.770332 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.770053    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.770332 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.770132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqdpl\" (UniqueName: \"kubernetes.io/projected/c5251119-9377-45db-a145-5c180ccdea47-kube-api-access-lqdpl\") pod \"payload-processing-e2e-ait-60243cce-69f485c788-bl85k\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") " pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.770332 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.770164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbm7m\" (UniqueName: \"kubernetes.io/projected/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-kube-api-access-qbm7m\") pod \"payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.770781 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.770761    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5251119-9377-45db-a145-5c180ccdea47-plugins-config-volume\") pod \"payload-processing-e2e-ait-60243cce-69f485c788-bl85k\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") " pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.770943 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.770922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.778043 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.778023    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqdpl\" (UniqueName: \"kubernetes.io/projected/c5251119-9377-45db-a145-5c180ccdea47-kube-api-access-lqdpl\") pod \"payload-processing-e2e-ait-60243cce-69f485c788-bl85k\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") " pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:26.778128 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.778111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbm7m\" (UniqueName: \"kubernetes.io/projected/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-kube-api-access-qbm7m\") pod \"payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.884788 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.884763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:26.928851 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:26.928823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:27.032806 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.032774    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"]
Jul 28 01:15:27.040003 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:15:27.039976    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90e16fe3_7fd6_4906_93f8_9ca6b06589cc.slice/crio-4793f0a1a4d21b46ada6c8274193b6718fb7ff5523179fac0e44a219df97c7be WatchSource:0}: Error finding container 4793f0a1a4d21b46ada6c8274193b6718fb7ff5523179fac0e44a219df97c7be: Status 404 returned error can't find the container with id 4793f0a1a4d21b46ada6c8274193b6718fb7ff5523179fac0e44a219df97c7be
Jul 28 01:15:27.074215 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.074191    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"]
Jul 28 01:15:27.075827 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:15:27.075799    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5251119_9377_45db_a145_5c180ccdea47.slice/crio-77e5a968b1df094c4200d780c70524b40b1b7f946d0cc52cff87b4208dbec99f WatchSource:0}: Error finding container 77e5a968b1df094c4200d780c70524b40b1b7f946d0cc52cff87b4208dbec99f: Status 404 returned error can't find the container with id 77e5a968b1df094c4200d780c70524b40b1b7f946d0cc52cff87b4208dbec99f
Jul 28 01:15:27.295488 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.295405    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k" event={"ID":"c5251119-9377-45db-a145-5c180ccdea47","Type":"ContainerStarted","Data":"a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed"}
Jul 28 01:15:27.295488 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.295468    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:27.295488 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.295487    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k" event={"ID":"c5251119-9377-45db-a145-5c180ccdea47","Type":"ContainerStarted","Data":"77e5a968b1df094c4200d780c70524b40b1b7f946d0cc52cff87b4208dbec99f"}
Jul 28 01:15:27.297001 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.296976    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj" event={"ID":"90e16fe3-7fd6-4906-93f8-9ca6b06589cc","Type":"ContainerStarted","Data":"f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5"}
Jul 28 01:15:27.297100 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.297019    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj" event={"ID":"90e16fe3-7fd6-4906-93f8-9ca6b06589cc","Type":"ContainerStarted","Data":"4793f0a1a4d21b46ada6c8274193b6718fb7ff5523179fac0e44a219df97c7be"}
Jul 28 01:15:27.297100 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.297056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:27.313461 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:27.313421    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k" podStartSLOduration=1.313408172 podStartE2EDuration="1.313408172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:27.312473278 +0000 UTC m=+1732.061612014" watchObservedRunningTime="2026-07-28 01:15:27.313408172 +0000 UTC m=+1732.062546897"
Jul 28 01:15:38.303795 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:38.303763    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:15:38.304178 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:38.303981    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:15:38.319804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:38.319751    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj" podStartSLOduration=12.319733888 podStartE2EDuration="12.319733888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:15:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:15:27.332125873 +0000 UTC m=+1732.081264594" watchObservedRunningTime="2026-07-28 01:15:38.319733888 +0000 UTC m=+1743.068872612"
Jul 28 01:15:38.438462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:15:38.438428    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:01.873365 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:01.873329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:01.903769 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:01.903735    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"]
Jul 28 01:16:01.903962 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:01.903933    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k" podUID="c5251119-9377-45db-a145-5c180ccdea47" containerName="payload-processing" containerID="cri-o://a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed" gracePeriod=30
Jul 28 01:16:01.910100 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:01.910073    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"]
Jul 28 01:16:01.911288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:01.911256    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj" podUID="90e16fe3-7fd6-4906-93f8-9ca6b06589cc" containerName="payload-pre-processing" containerID="cri-o://f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5" gracePeriod=30
Jul 28 01:16:01.943253 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:01.943207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:02.044363 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.044323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:02.180276 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.180253    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:16:02.182693 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.182676    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:16:02.268163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.268131    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-plugins-config-volume\") pod \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") "
Jul 28 01:16:02.268358 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.268179    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5251119-9377-45db-a145-5c180ccdea47-plugins-config-volume\") pod \"c5251119-9377-45db-a145-5c180ccdea47\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") "
Jul 28 01:16:02.268358 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.268195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqdpl\" (UniqueName: \"kubernetes.io/projected/c5251119-9377-45db-a145-5c180ccdea47-kube-api-access-lqdpl\") pod \"c5251119-9377-45db-a145-5c180ccdea47\" (UID: \"c5251119-9377-45db-a145-5c180ccdea47\") "
Jul 28 01:16:02.268358 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.268216    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbm7m\" (UniqueName: \"kubernetes.io/projected/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-kube-api-access-qbm7m\") pod \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\" (UID: \"90e16fe3-7fd6-4906-93f8-9ca6b06589cc\") "
Jul 28 01:16:02.268633 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.268605    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90e16fe3-7fd6-4906-93f8-9ca6b06589cc" (UID: "90e16fe3-7fd6-4906-93f8-9ca6b06589cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:02.268705 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.268611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5251119-9377-45db-a145-5c180ccdea47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c5251119-9377-45db-a145-5c180ccdea47" (UID: "c5251119-9377-45db-a145-5c180ccdea47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:02.270360 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.270333    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5251119-9377-45db-a145-5c180ccdea47-kube-api-access-lqdpl" (OuterVolumeSpecName: "kube-api-access-lqdpl") pod "c5251119-9377-45db-a145-5c180ccdea47" (UID: "c5251119-9377-45db-a145-5c180ccdea47"). InnerVolumeSpecName "kube-api-access-lqdpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:02.270360 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.270357    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-kube-api-access-qbm7m" (OuterVolumeSpecName: "kube-api-access-qbm7m") pod "90e16fe3-7fd6-4906-93f8-9ca6b06589cc" (UID: "90e16fe3-7fd6-4906-93f8-9ca6b06589cc"). InnerVolumeSpecName "kube-api-access-qbm7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:02.369527 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.369495    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.369527 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.369520    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5251119-9377-45db-a145-5c180ccdea47-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.369527 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.369529    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqdpl\" (UniqueName: \"kubernetes.io/projected/c5251119-9377-45db-a145-5c180ccdea47-kube-api-access-lqdpl\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.369748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.369538    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbm7m\" (UniqueName: \"kubernetes.io/projected/90e16fe3-7fd6-4906-93f8-9ca6b06589cc-kube-api-access-qbm7m\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:02.450761 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.450667    2576 generic.go:356] "Generic (PLEG): container finished" podID="90e16fe3-7fd6-4906-93f8-9ca6b06589cc" containerID="f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5" exitCode=0
Jul 28 01:16:02.450761 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.450748    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"
Jul 28 01:16:02.450977 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.450792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj" event={"ID":"90e16fe3-7fd6-4906-93f8-9ca6b06589cc","Type":"ContainerDied","Data":"f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5"}
Jul 28 01:16:02.450977 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.450840    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj" event={"ID":"90e16fe3-7fd6-4906-93f8-9ca6b06589cc","Type":"ContainerDied","Data":"4793f0a1a4d21b46ada6c8274193b6718fb7ff5523179fac0e44a219df97c7be"}
Jul 28 01:16:02.450977 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.450864    2576 scope.go:117] "RemoveContainer" containerID="f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5"
Jul 28 01:16:02.452177 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.452153    2576 generic.go:356] "Generic (PLEG): container finished" podID="c5251119-9377-45db-a145-5c180ccdea47" containerID="a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed" exitCode=0
Jul 28 01:16:02.452310 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.452196    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"
Jul 28 01:16:02.452310 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.452200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k" event={"ID":"c5251119-9377-45db-a145-5c180ccdea47","Type":"ContainerDied","Data":"a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed"}
Jul 28 01:16:02.452310 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.452255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k" event={"ID":"c5251119-9377-45db-a145-5c180ccdea47","Type":"ContainerDied","Data":"77e5a968b1df094c4200d780c70524b40b1b7f946d0cc52cff87b4208dbec99f"}
Jul 28 01:16:02.459193 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.459176    2576 scope.go:117] "RemoveContainer" containerID="f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5"
Jul 28 01:16:02.459515 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:16:02.459492    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5\": container with ID starting with f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5 not found: ID does not exist" containerID="f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5"
Jul 28 01:16:02.459606 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.459525    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5"} err="failed to get container status \"f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5\": rpc error: code = NotFound desc = could not find container \"f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5\": container with ID starting with f88960756c3fc16375c0c91e4ffe85deb7089197091024a2d6539eb112c417a5 not found: ID does not exist"
Jul 28 01:16:02.459606 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.459549    2576 scope.go:117] "RemoveContainer" containerID="a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed"
Jul 28 01:16:02.467066 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.467048    2576 scope.go:117] "RemoveContainer" containerID="a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed"
Jul 28 01:16:02.467334 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:16:02.467314    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed\": container with ID starting with a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed not found: ID does not exist" containerID="a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed"
Jul 28 01:16:02.467382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.467343    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed"} err="failed to get container status \"a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed\": rpc error: code = NotFound desc = could not find container \"a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed\": container with ID starting with a3aecdd13ce9c66ffb6ac5d30fb507f3790faafd0f0726f2890da23bd3ec52ed not found: ID does not exist"
Jul 28 01:16:02.477858 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.477833    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"]
Jul 28 01:16:02.481519 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.481498    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-60243cce-69f485c788-bl85k"]
Jul 28 01:16:02.491685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.491663    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"]
Jul 28 01:16:02.496164 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:02.496144    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-60243cce-5dd8f6fb44-lkcvj"]
Jul 28 01:16:03.751643 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:03.751600    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90e16fe3-7fd6-4906-93f8-9ca6b06589cc" path="/var/lib/kubelet/pods/90e16fe3-7fd6-4906-93f8-9ca6b06589cc/volumes"
Jul 28 01:16:03.752134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:03.752095    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5251119-9377-45db-a145-5c180ccdea47" path="/var/lib/kubelet/pods/c5251119-9377-45db-a145-5c180ccdea47/volumes"
Jul 28 01:16:06.525200 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:06.525165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:07.802307 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:07.802274    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:07.963947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:07.963909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:08.186134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:08.186103    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:08.531926 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:08.531824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:09.163689 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:09.163652    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:10.856323 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:10.856282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:10.952547 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:10.952519    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:11.135464 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.135385    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:11.446322 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.446234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:11.684542 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.684513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:11.787531 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787460    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"]
Jul 28 01:16:11.787787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787775    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90e16fe3-7fd6-4906-93f8-9ca6b06589cc" containerName="payload-pre-processing"
Jul 28 01:16:11.787830 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787789    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="90e16fe3-7fd6-4906-93f8-9ca6b06589cc" containerName="payload-pre-processing"
Jul 28 01:16:11.787830 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787806    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5251119-9377-45db-a145-5c180ccdea47" containerName="payload-processing"
Jul 28 01:16:11.787830 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787812    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5251119-9377-45db-a145-5c180ccdea47" containerName="payload-processing"
Jul 28 01:16:11.787932 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787872    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5251119-9377-45db-a145-5c180ccdea47" containerName="payload-processing"
Jul 28 01:16:11.787932 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.787883    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="90e16fe3-7fd6-4906-93f8-9ca6b06589cc" containerName="payload-pre-processing"
Jul 28 01:16:11.790394 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.790375    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:11.792516 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.792491    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:11.792987 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.792971    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-f6442e46\""
Jul 28 01:16:11.793050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.792971    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-f6442e46-dockercfg-r8p9z\""
Jul 28 01:16:11.801263 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.801241    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"]
Jul 28 01:16:11.830813 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.830792    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"]
Jul 28 01:16:11.834241 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.834212    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:11.842069 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.842044    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"]
Jul 28 01:16:11.948983 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.948928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mtmv\" (UniqueName: \"kubernetes.io/projected/6652d5bd-1a20-484e-8cc2-5d3e94d36283-kube-api-access-9mtmv\") pod \"payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:11.949364 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.949111    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6652d5bd-1a20-484e-8cc2-5d3e94d36283-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:11.949364 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.949141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ece751-bbcc-40be-8db7-44abbee9c68b-plugins-config-volume\") pod \"payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") " pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:11.949364 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:11.949162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tct5p\" (UniqueName: \"kubernetes.io/projected/05ece751-bbcc-40be-8db7-44abbee9c68b-kube-api-access-tct5p\") pod \"payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") " pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.049917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.049827    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6652d5bd-1a20-484e-8cc2-5d3e94d36283-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:12.049917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.049864    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ece751-bbcc-40be-8db7-44abbee9c68b-plugins-config-volume\") pod \"payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") " pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.049917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.049883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tct5p\" (UniqueName: \"kubernetes.io/projected/05ece751-bbcc-40be-8db7-44abbee9c68b-kube-api-access-tct5p\") pod \"payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") " pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.050187 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.049945    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mtmv\" (UniqueName: \"kubernetes.io/projected/6652d5bd-1a20-484e-8cc2-5d3e94d36283-kube-api-access-9mtmv\") pod \"payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:12.050654 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.050632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6652d5bd-1a20-484e-8cc2-5d3e94d36283-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:12.050718 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.050660    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ece751-bbcc-40be-8db7-44abbee9c68b-plugins-config-volume\") pod \"payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") " pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.057846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.057825    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tct5p\" (UniqueName: \"kubernetes.io/projected/05ece751-bbcc-40be-8db7-44abbee9c68b-kube-api-access-tct5p\") pod \"payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") " pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.057965 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.057847    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mtmv\" (UniqueName: \"kubernetes.io/projected/6652d5bd-1a20-484e-8cc2-5d3e94d36283-kube-api-access-9mtmv\") pod \"payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:12.101642 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.101618    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:12.144960 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.144924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.247755 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.247676    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"]
Jul 28 01:16:12.250266 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:16:12.250239    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6652d5bd_1a20_484e_8cc2_5d3e94d36283.slice/crio-636ee8d40baf63029746588520b18b706efdd999d7e0c27a899803be1c0d6739 WatchSource:0}: Error finding container 636ee8d40baf63029746588520b18b706efdd999d7e0c27a899803be1c0d6739: Status 404 returned error can't find the container with id 636ee8d40baf63029746588520b18b706efdd999d7e0c27a899803be1c0d6739
Jul 28 01:16:12.286571 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.286499    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"]
Jul 28 01:16:12.288934 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:16:12.288905    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05ece751_bbcc_40be_8db7_44abbee9c68b.slice/crio-5be71d884b7fce624181d312011335814bd17da4cfaa53d6ba8b7f906a5e5b6c WatchSource:0}: Error finding container 5be71d884b7fce624181d312011335814bd17da4cfaa53d6ba8b7f906a5e5b6c: Status 404 returned error can't find the container with id 5be71d884b7fce624181d312011335814bd17da4cfaa53d6ba8b7f906a5e5b6c
Jul 28 01:16:12.501012 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.500977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2" event={"ID":"05ece751-bbcc-40be-8db7-44abbee9c68b","Type":"ContainerStarted","Data":"0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5"}
Jul 28 01:16:12.501199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.501040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2" event={"ID":"05ece751-bbcc-40be-8db7-44abbee9c68b","Type":"ContainerStarted","Data":"5be71d884b7fce624181d312011335814bd17da4cfaa53d6ba8b7f906a5e5b6c"}
Jul 28 01:16:12.501199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.501069    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:12.502402 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.502379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw" event={"ID":"6652d5bd-1a20-484e-8cc2-5d3e94d36283","Type":"ContainerStarted","Data":"909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a"}
Jul 28 01:16:12.502510 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.502429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw" event={"ID":"6652d5bd-1a20-484e-8cc2-5d3e94d36283","Type":"ContainerStarted","Data":"636ee8d40baf63029746588520b18b706efdd999d7e0c27a899803be1c0d6739"}
Jul 28 01:16:12.502510 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.502464    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:12.517707 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.517660    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2" podStartSLOduration=1.51764712 podStartE2EDuration="1.51764712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:12.516166602 +0000 UTC m=+1777.265305329" watchObservedRunningTime="2026-07-28 01:16:12.51764712 +0000 UTC m=+1777.266785847"
Jul 28 01:16:12.531404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:12.531363    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw" podStartSLOduration=1.53135048 podStartE2EDuration="1.53135048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:16:12.530514776 +0000 UTC m=+1777.279653503" watchObservedRunningTime="2026-07-28 01:16:12.53135048 +0000 UTC m=+1777.280489205"
Jul 28 01:16:23.509799 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:23.509764    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:23.510275 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:23.509875    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:46.290396 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.290353    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:46.316617 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.316585    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"]
Jul 28 01:16:46.317067 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.317019    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2" podUID="05ece751-bbcc-40be-8db7-44abbee9c68b" containerName="payload-processing" containerID="cri-o://0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5" gracePeriod=30
Jul 28 01:16:46.324043 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.324019    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"]
Jul 28 01:16:46.324211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.324192    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw" podUID="6652d5bd-1a20-484e-8cc2-5d3e94d36283" containerName="payload-pre-processing" containerID="cri-o://909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a" gracePeriod=30
Jul 28 01:16:46.354506 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.354476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:46.410213 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.410185    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:46.581926 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.581901    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:46.595151 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.595127    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:46.632039 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.632010    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tct5p\" (UniqueName: \"kubernetes.io/projected/05ece751-bbcc-40be-8db7-44abbee9c68b-kube-api-access-tct5p\") pod \"05ece751-bbcc-40be-8db7-44abbee9c68b\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") "
Jul 28 01:16:46.632169 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.632056    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ece751-bbcc-40be-8db7-44abbee9c68b-plugins-config-volume\") pod \"05ece751-bbcc-40be-8db7-44abbee9c68b\" (UID: \"05ece751-bbcc-40be-8db7-44abbee9c68b\") "
Jul 28 01:16:46.632383 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.632360    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05ece751-bbcc-40be-8db7-44abbee9c68b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "05ece751-bbcc-40be-8db7-44abbee9c68b" (UID: "05ece751-bbcc-40be-8db7-44abbee9c68b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:46.633982 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.633959    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05ece751-bbcc-40be-8db7-44abbee9c68b-kube-api-access-tct5p" (OuterVolumeSpecName: "kube-api-access-tct5p") pod "05ece751-bbcc-40be-8db7-44abbee9c68b" (UID: "05ece751-bbcc-40be-8db7-44abbee9c68b"). InnerVolumeSpecName "kube-api-access-tct5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:46.661028 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.661001    2576 generic.go:356] "Generic (PLEG): container finished" podID="6652d5bd-1a20-484e-8cc2-5d3e94d36283" containerID="909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a" exitCode=0
Jul 28 01:16:46.661141 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.661044    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"
Jul 28 01:16:46.661141 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.661080    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw" event={"ID":"6652d5bd-1a20-484e-8cc2-5d3e94d36283","Type":"ContainerDied","Data":"909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a"}
Jul 28 01:16:46.661141 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.661122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw" event={"ID":"6652d5bd-1a20-484e-8cc2-5d3e94d36283","Type":"ContainerDied","Data":"636ee8d40baf63029746588520b18b706efdd999d7e0c27a899803be1c0d6739"}
Jul 28 01:16:46.661345 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.661145    2576 scope.go:117] "RemoveContainer" containerID="909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a"
Jul 28 01:16:46.662536 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.662511    2576 generic.go:356] "Generic (PLEG): container finished" podID="05ece751-bbcc-40be-8db7-44abbee9c68b" containerID="0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5" exitCode=0
Jul 28 01:16:46.662641 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.662551    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"
Jul 28 01:16:46.662641 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.662595    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2" event={"ID":"05ece751-bbcc-40be-8db7-44abbee9c68b","Type":"ContainerDied","Data":"0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5"}
Jul 28 01:16:46.662731 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.662637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2" event={"ID":"05ece751-bbcc-40be-8db7-44abbee9c68b","Type":"ContainerDied","Data":"5be71d884b7fce624181d312011335814bd17da4cfaa53d6ba8b7f906a5e5b6c"}
Jul 28 01:16:46.670521 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.670506    2576 scope.go:117] "RemoveContainer" containerID="909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a"
Jul 28 01:16:46.670771 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:16:46.670750    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a\": container with ID starting with 909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a not found: ID does not exist" containerID="909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a"
Jul 28 01:16:46.670848 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.670782    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a"} err="failed to get container status \"909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a\": rpc error: code = NotFound desc = could not find container \"909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a\": container with ID starting with 909e25f519a18d47269db6bd348feb29d1883031bd0abd3873cb2497a4793f1a not found: ID does not exist"
Jul 28 01:16:46.670848 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.670804    2576 scope.go:117] "RemoveContainer" containerID="0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5"
Jul 28 01:16:46.678331 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.678216    2576 scope.go:117] "RemoveContainer" containerID="0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5"
Jul 28 01:16:46.678589 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:16:46.678570    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5\": container with ID starting with 0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5 not found: ID does not exist" containerID="0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5"
Jul 28 01:16:46.678650 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.678595    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5"} err="failed to get container status \"0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5\": rpc error: code = NotFound desc = could not find container \"0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5\": container with ID starting with 0c8c397cb8ef1b0928a5c87dfe6bb6d7b97ee939a190c45210961329ca78dec5 not found: ID does not exist"
Jul 28 01:16:46.685321 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.685301    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"]
Jul 28 01:16:46.689055 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.689035    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-f6442e46-5ff685779d-g2wh2"]
Jul 28 01:16:46.732927 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.732906    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9mtmv\" (UniqueName: \"kubernetes.io/projected/6652d5bd-1a20-484e-8cc2-5d3e94d36283-kube-api-access-9mtmv\") pod \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") "
Jul 28 01:16:46.732997 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.732950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6652d5bd-1a20-484e-8cc2-5d3e94d36283-plugins-config-volume\") pod \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\" (UID: \"6652d5bd-1a20-484e-8cc2-5d3e94d36283\") "
Jul 28 01:16:46.733106 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.733094    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tct5p\" (UniqueName: \"kubernetes.io/projected/05ece751-bbcc-40be-8db7-44abbee9c68b-kube-api-access-tct5p\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.733145 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.733107    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05ece751-bbcc-40be-8db7-44abbee9c68b-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.733374 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.733347    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6652d5bd-1a20-484e-8cc2-5d3e94d36283-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6652d5bd-1a20-484e-8cc2-5d3e94d36283" (UID: "6652d5bd-1a20-484e-8cc2-5d3e94d36283"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:16:46.734771 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.734755    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6652d5bd-1a20-484e-8cc2-5d3e94d36283-kube-api-access-9mtmv" (OuterVolumeSpecName: "kube-api-access-9mtmv") pod "6652d5bd-1a20-484e-8cc2-5d3e94d36283" (UID: "6652d5bd-1a20-484e-8cc2-5d3e94d36283"). InnerVolumeSpecName "kube-api-access-9mtmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:16:46.834084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.834015    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mtmv\" (UniqueName: \"kubernetes.io/projected/6652d5bd-1a20-484e-8cc2-5d3e94d36283-kube-api-access-9mtmv\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.834084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.834048    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6652d5bd-1a20-484e-8cc2-5d3e94d36283-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:16:46.981235 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.981204    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"]
Jul 28 01:16:46.987002 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:46.986974    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-f6442e46-6845fd5fbd-42mdw"]
Jul 28 01:16:47.751325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:47.751212    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05ece751-bbcc-40be-8db7-44abbee9c68b" path="/var/lib/kubelet/pods/05ece751-bbcc-40be-8db7-44abbee9c68b/volumes"
Jul 28 01:16:47.751814 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:47.751783    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6652d5bd-1a20-484e-8cc2-5d3e94d36283" path="/var/lib/kubelet/pods/6652d5bd-1a20-484e-8cc2-5d3e94d36283/volumes"
Jul 28 01:16:50.772598 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:50.772564    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:50.930292 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:50.930255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:51.157746 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:51.157714    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:51.605829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:51.605787    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:51.709609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:51.709578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:51.805465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:51.805432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:53.241019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:53.240988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:58.848365 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:58.848326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:58.957241 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:58.957188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:59.049971 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:59.049940    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:59.147878 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:59.147810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:16:59.522795 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:16:59.522710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:25.455798 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:25.455763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:31.805054 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:31.805020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:31.908238 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:31.908187    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:32.005584 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:32.005548    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:47.869880 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:47.869843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:53.436280 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:53.436248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:53.499709 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:53.499682    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:53.597728 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:53.597704    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:17:53.686568 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:17:53.686501    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:04.148974 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:04.148934    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:04.907407 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:04.907368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:11.346768 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:11.346730    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:12.529163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:12.529124    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:17.214473 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:17.214436    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:17.574479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:17.574447    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:17.674355 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:17.674320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:17.774085 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:17.774055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:19.390256 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:19.390212    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:25.357629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:25.357583    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:25.460509 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:25.460476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:26.020242 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:26.020189    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:32.211283 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:32.211244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:32.276392 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:32.276350    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:32.431767 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:32.431740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:32.734376 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:32.734339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:32.920861 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:32.920828    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:35.375677 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:35.375642    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:35.476275 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:35.476214    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:35.652316 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:35.652239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:35.855972 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:35.855939    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:36.269243 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.269197    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:36.366568 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.366534    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"]
Jul 28 01:18:36.366937 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.366920    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05ece751-bbcc-40be-8db7-44abbee9c68b" containerName="payload-processing"
Jul 28 01:18:36.367028 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.366942    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05ece751-bbcc-40be-8db7-44abbee9c68b" containerName="payload-processing"
Jul 28 01:18:36.367028 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.366984    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6652d5bd-1a20-484e-8cc2-5d3e94d36283" containerName="payload-pre-processing"
Jul 28 01:18:36.367028 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.366994    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6652d5bd-1a20-484e-8cc2-5d3e94d36283" containerName="payload-pre-processing"
Jul 28 01:18:36.367264 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.367069    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="05ece751-bbcc-40be-8db7-44abbee9c68b" containerName="payload-processing"
Jul 28 01:18:36.367264 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.367082    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6652d5bd-1a20-484e-8cc2-5d3e94d36283" containerName="payload-pre-processing"
Jul 28 01:18:36.370040 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.370021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.370440 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.370411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:36.372666 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.372649    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-730aaa5d\""
Jul 28 01:18:36.379053 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.379030    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"]
Jul 28 01:18:36.405512 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.405488    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"]
Jul 28 01:18:36.408685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.408670    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.411365 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.411346    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-730aaa5d-dockercfg-8whbg\""
Jul 28 01:18:36.417380 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.417363    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"]
Jul 28 01:18:36.445858 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.445836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.445981 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.445878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cxb\" (UniqueName: \"kubernetes.io/projected/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-kube-api-access-j8cxb\") pod \"payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.546554 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.546489    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc2264-517d-416f-a44a-08e3dc32a4ea-plugins-config-volume\") pod \"payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") " pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.546554 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.546532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww6z8\" (UniqueName: \"kubernetes.io/projected/2bfc2264-517d-416f-a44a-08e3dc32a4ea-kube-api-access-ww6z8\") pod \"payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") " pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.546717 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.546555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cxb\" (UniqueName: \"kubernetes.io/projected/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-kube-api-access-j8cxb\") pod \"payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.546717 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.546673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.547282 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.547263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.557897 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.557871    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cxb\" (UniqueName: \"kubernetes.io/projected/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-kube-api-access-j8cxb\") pod \"payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.648073 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.648045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww6z8\" (UniqueName: \"kubernetes.io/projected/2bfc2264-517d-416f-a44a-08e3dc32a4ea-kube-api-access-ww6z8\") pod \"payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") " pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.648200 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.648139    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc2264-517d-416f-a44a-08e3dc32a4ea-plugins-config-volume\") pod \"payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") " pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.648817 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.648798    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc2264-517d-416f-a44a-08e3dc32a4ea-plugins-config-volume\") pod \"payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") " pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.661442 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.661418    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww6z8\" (UniqueName: \"kubernetes.io/projected/2bfc2264-517d-416f-a44a-08e3dc32a4ea-kube-api-access-ww6z8\") pod \"payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") " pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.681361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.681337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:36.719134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.719096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:36.819015 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.818988    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"]
Jul 28 01:18:36.821021 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:18:36.820991    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb256e36_f6fb_465a_b5ad_ff3e29d49e26.slice/crio-426b0411a80810efad3b95592b1c5f48982106b5b4af5f2511fb1a4b118aa666 WatchSource:0}: Error finding container 426b0411a80810efad3b95592b1c5f48982106b5b4af5f2511fb1a4b118aa666: Status 404 returned error can't find the container with id 426b0411a80810efad3b95592b1c5f48982106b5b4af5f2511fb1a4b118aa666
Jul 28 01:18:36.879506 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:36.879481    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"]
Jul 28 01:18:36.882297 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:18:36.882268    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfc2264_517d_416f_a44a_08e3dc32a4ea.slice/crio-7cab171ff86a7718cdfd8297bf7705f37bc2bbd6a6589d540d5d1375435163bd WatchSource:0}: Error finding container 7cab171ff86a7718cdfd8297bf7705f37bc2bbd6a6589d540d5d1375435163bd: Status 404 returned error can't find the container with id 7cab171ff86a7718cdfd8297bf7705f37bc2bbd6a6589d540d5d1375435163bd
Jul 28 01:18:37.131084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.130981    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5" event={"ID":"fb256e36-f6fb-465a-b5ad-ff3e29d49e26","Type":"ContainerStarted","Data":"83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118"}
Jul 28 01:18:37.131084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.131048    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5" event={"ID":"fb256e36-f6fb-465a-b5ad-ff3e29d49e26","Type":"ContainerStarted","Data":"426b0411a80810efad3b95592b1c5f48982106b5b4af5f2511fb1a4b118aa666"}
Jul 28 01:18:37.131084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.131075    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:37.132479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.132450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs" event={"ID":"2bfc2264-517d-416f-a44a-08e3dc32a4ea","Type":"ContainerStarted","Data":"cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a"}
Jul 28 01:18:37.132592 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.132493    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:37.132592 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.132509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs" event={"ID":"2bfc2264-517d-416f-a44a-08e3dc32a4ea","Type":"ContainerStarted","Data":"7cab171ff86a7718cdfd8297bf7705f37bc2bbd6a6589d540d5d1375435163bd"}
Jul 28 01:18:37.148566 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.148145    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5" podStartSLOduration=1.148129254 podStartE2EDuration="1.148129254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:18:37.147778828 +0000 UTC m=+1921.896917553" watchObservedRunningTime="2026-07-28 01:18:37.148129254 +0000 UTC m=+1921.897267978"
Jul 28 01:18:37.166165 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:37.166108    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs" podStartSLOduration=1.166090822 podStartE2EDuration="1.166090822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:18:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:18:37.1644189 +0000 UTC m=+1921.913557617" watchObservedRunningTime="2026-07-28 01:18:37.166090822 +0000 UTC m=+1921.915229563"
Jul 28 01:18:39.847740 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:39.847695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:40.473636 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:40.473602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:40.569507 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:40.569478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:40.875111 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:40.875078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:46.595832 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:46.595790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:46.662324 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:46.662292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:48.138956 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:48.138924    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:48.139375 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:48.139074    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:53.577342 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:53.577301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:57.535832 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.535795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:57.562666 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.562634    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"]
Jul 28 01:18:57.562928 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.562869    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs" podUID="2bfc2264-517d-416f-a44a-08e3dc32a4ea" containerName="payload-processing" containerID="cri-o://cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a" gracePeriod=30
Jul 28 01:18:57.568887 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.568862    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"]
Jul 28 01:18:57.569273 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.569241    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5" podUID="fb256e36-f6fb-465a-b5ad-ff3e29d49e26" containerName="payload-pre-processing" containerID="cri-o://83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118" gracePeriod=30
Jul 28 01:18:57.635787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.635758    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:57.691605 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.691569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:18:57.808363 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.808342    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:57.835005 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.834975    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:57.839031 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.839008    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc2264-517d-416f-a44a-08e3dc32a4ea-plugins-config-volume\") pod \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") "
Jul 28 01:18:57.839141 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.839037    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww6z8\" (UniqueName: \"kubernetes.io/projected/2bfc2264-517d-416f-a44a-08e3dc32a4ea-kube-api-access-ww6z8\") pod \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\" (UID: \"2bfc2264-517d-416f-a44a-08e3dc32a4ea\") "
Jul 28 01:18:57.839338 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.839312    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bfc2264-517d-416f-a44a-08e3dc32a4ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bfc2264-517d-416f-a44a-08e3dc32a4ea" (UID: "2bfc2264-517d-416f-a44a-08e3dc32a4ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:18:57.839429 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.839411    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bfc2264-517d-416f-a44a-08e3dc32a4ea-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:18:57.841077 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.841055    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bfc2264-517d-416f-a44a-08e3dc32a4ea-kube-api-access-ww6z8" (OuterVolumeSpecName: "kube-api-access-ww6z8") pod "2bfc2264-517d-416f-a44a-08e3dc32a4ea" (UID: "2bfc2264-517d-416f-a44a-08e3dc32a4ea"). InnerVolumeSpecName "kube-api-access-ww6z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:18:57.940400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.940374    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8cxb\" (UniqueName: \"kubernetes.io/projected/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-kube-api-access-j8cxb\") pod \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") "
Jul 28 01:18:57.940554 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.940436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-plugins-config-volume\") pod \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\" (UID: \"fb256e36-f6fb-465a-b5ad-ff3e29d49e26\") "
Jul 28 01:18:57.940689 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.940670    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww6z8\" (UniqueName: \"kubernetes.io/projected/2bfc2264-517d-416f-a44a-08e3dc32a4ea-kube-api-access-ww6z8\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:18:57.940832 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.940815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fb256e36-f6fb-465a-b5ad-ff3e29d49e26" (UID: "fb256e36-f6fb-465a-b5ad-ff3e29d49e26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:18:57.942376 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:57.942349    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-kube-api-access-j8cxb" (OuterVolumeSpecName: "kube-api-access-j8cxb") pod "fb256e36-f6fb-465a-b5ad-ff3e29d49e26" (UID: "fb256e36-f6fb-465a-b5ad-ff3e29d49e26"). InnerVolumeSpecName "kube-api-access-j8cxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:18:58.041843 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.041772    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8cxb\" (UniqueName: \"kubernetes.io/projected/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-kube-api-access-j8cxb\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:18:58.041843 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.041796    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fb256e36-f6fb-465a-b5ad-ff3e29d49e26-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:18:58.231572 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.231537    2576 generic.go:356] "Generic (PLEG): container finished" podID="fb256e36-f6fb-465a-b5ad-ff3e29d49e26" containerID="83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118" exitCode=0
Jul 28 01:18:58.231748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.231583    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"
Jul 28 01:18:58.231748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.231611    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5" event={"ID":"fb256e36-f6fb-465a-b5ad-ff3e29d49e26","Type":"ContainerDied","Data":"83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118"}
Jul 28 01:18:58.231748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.231657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5" event={"ID":"fb256e36-f6fb-465a-b5ad-ff3e29d49e26","Type":"ContainerDied","Data":"426b0411a80810efad3b95592b1c5f48982106b5b4af5f2511fb1a4b118aa666"}
Jul 28 01:18:58.231748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.231680    2576 scope.go:117] "RemoveContainer" containerID="83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118"
Jul 28 01:18:58.232933 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.232914    2576 generic.go:356] "Generic (PLEG): container finished" podID="2bfc2264-517d-416f-a44a-08e3dc32a4ea" containerID="cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a" exitCode=0
Jul 28 01:18:58.233046 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.232939    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs" event={"ID":"2bfc2264-517d-416f-a44a-08e3dc32a4ea","Type":"ContainerDied","Data":"cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a"}
Jul 28 01:18:58.233046 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.232964    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"
Jul 28 01:18:58.233046 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.232987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs" event={"ID":"2bfc2264-517d-416f-a44a-08e3dc32a4ea","Type":"ContainerDied","Data":"7cab171ff86a7718cdfd8297bf7705f37bc2bbd6a6589d540d5d1375435163bd"}
Jul 28 01:18:58.240099 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.240079    2576 scope.go:117] "RemoveContainer" containerID="83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118"
Jul 28 01:18:58.240364 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:18:58.240341    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118\": container with ID starting with 83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118 not found: ID does not exist" containerID="83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118"
Jul 28 01:18:58.240475 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.240374    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118"} err="failed to get container status \"83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118\": rpc error: code = NotFound desc = could not find container \"83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118\": container with ID starting with 83078559a7ecc566d3476acd129eb01d04b8dacfa690344d8949af5375df8118 not found: ID does not exist"
Jul 28 01:18:58.240475 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.240398    2576 scope.go:117] "RemoveContainer" containerID="cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a"
Jul 28 01:18:58.248282 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.248262    2576 scope.go:117] "RemoveContainer" containerID="cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a"
Jul 28 01:18:58.248585 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:18:58.248566    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a\": container with ID starting with cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a not found: ID does not exist" containerID="cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a"
Jul 28 01:18:58.248655 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.248593    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a"} err="failed to get container status \"cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a\": rpc error: code = NotFound desc = could not find container \"cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a\": container with ID starting with cd32d30a9443dae45cd878daa7dcbac0c0b3895eb34e78cf4897c277a989488a not found: ID does not exist"
Jul 28 01:18:58.259110 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.259091    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"]
Jul 28 01:18:58.263055 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.263036    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-730aaa5d-85c6868c96-8zll5"]
Jul 28 01:18:58.273784 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.273754    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"]
Jul 28 01:18:58.276973 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:58.276951    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-730aaa5d-75f4c8d646-dp8xs"]
Jul 28 01:18:59.752237 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:59.752201    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bfc2264-517d-416f-a44a-08e3dc32a4ea" path="/var/lib/kubelet/pods/2bfc2264-517d-416f-a44a-08e3dc32a4ea/volumes"
Jul 28 01:18:59.752607 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:18:59.752584    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb256e36-f6fb-465a-b5ad-ff3e29d49e26" path="/var/lib/kubelet/pods/fb256e36-f6fb-465a-b5ad-ff3e29d49e26/volumes"
Jul 28 01:19:02.020321 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:02.020288    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:02.406327 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:02.406281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:08.271342 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:08.271311    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:08.339814 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:08.339773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:08.534275 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:08.534199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:08.729603 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:08.729567    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:08.844402 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:08.844359    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:08.940987 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:08.940953    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:09.044296 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:09.044211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:15.203491 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:15.203459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:15.273205 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:15.273177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:15.358716 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:15.358684    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:15.599237 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:15.599203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:21.459458 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:21.459418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:21.516439 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:21.516407    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:28.253606 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:28.253526    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:28.557210 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:28.557180    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:28.786835 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:28.786805    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:31.453704 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:31.453669    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:31.543297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:31.543258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:31.627499 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:31.627467    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:32.085616 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.085589    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:32.189429 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.189384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:32.190166 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190141    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"]
Jul 28 01:19:32.190544 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190527    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bfc2264-517d-416f-a44a-08e3dc32a4ea" containerName="payload-processing"
Jul 28 01:19:32.190629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190546    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bfc2264-517d-416f-a44a-08e3dc32a4ea" containerName="payload-processing"
Jul 28 01:19:32.190629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190566    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb256e36-f6fb-465a-b5ad-ff3e29d49e26" containerName="payload-pre-processing"
Jul 28 01:19:32.190629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190575    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb256e36-f6fb-465a-b5ad-ff3e29d49e26" containerName="payload-pre-processing"
Jul 28 01:19:32.190794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190669    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb256e36-f6fb-465a-b5ad-ff3e29d49e26" containerName="payload-pre-processing"
Jul 28 01:19:32.190794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.190687    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bfc2264-517d-416f-a44a-08e3dc32a4ea" containerName="payload-processing"
Jul 28 01:19:32.193313 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.193296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.198915 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.198897    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-7ce60076-dockercfg-sgh94\""
Jul 28 01:19:32.199361 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.199341    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-7ce60076\""
Jul 28 01:19:32.208006 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.207987    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"]
Jul 28 01:19:32.230761 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.230734    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"]
Jul 28 01:19:32.234171 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.234157    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.239048 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.239025    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1e983d-d048-47d2-be21-339a4d32279c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.239167 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.239126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2sv\" (UniqueName: \"kubernetes.io/projected/ca1e983d-d048-47d2-be21-339a4d32279c-kube-api-access-sq2sv\") pod \"payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.244211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.244181    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"]
Jul 28 01:19:32.339862 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.339784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa9bb6-bafd-4042-bc18-79d8573646f4-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") " pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.339862 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.339829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2sv\" (UniqueName: \"kubernetes.io/projected/ca1e983d-d048-47d2-be21-339a4d32279c-kube-api-access-sq2sv\") pod \"payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.339862 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.339851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9shn\" (UniqueName: \"kubernetes.io/projected/73aa9bb6-bafd-4042-bc18-79d8573646f4-kube-api-access-q9shn\") pod \"payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") " pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.340084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.339906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1e983d-d048-47d2-be21-339a4d32279c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.340607 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.340588    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1e983d-d048-47d2-be21-339a4d32279c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.348249 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.348205    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2sv\" (UniqueName: \"kubernetes.io/projected/ca1e983d-d048-47d2-be21-339a4d32279c-kube-api-access-sq2sv\") pod \"payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.440577 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.440544    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa9bb6-bafd-4042-bc18-79d8573646f4-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") " pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.440708 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.440591    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9shn\" (UniqueName: \"kubernetes.io/projected/73aa9bb6-bafd-4042-bc18-79d8573646f4-kube-api-access-q9shn\") pod \"payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") " pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.441287 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.441266    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa9bb6-bafd-4042-bc18-79d8573646f4-plugins-config-volume\") pod \"payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") " pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.453168 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.453145    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9shn\" (UniqueName: \"kubernetes.io/projected/73aa9bb6-bafd-4042-bc18-79d8573646f4-kube-api-access-q9shn\") pod \"payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") " pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.503062 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.503032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:32.544834 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.544806    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:32.646543 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.646510    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"]
Jul 28 01:19:32.648610 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:19:32.648579    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca1e983d_d048_47d2_be21_339a4d32279c.slice/crio-586f523f0e40de3456da628654b529402a4a83951def37f9da2d8a4b466b9b41 WatchSource:0}: Error finding container 586f523f0e40de3456da628654b529402a4a83951def37f9da2d8a4b466b9b41: Status 404 returned error can't find the container with id 586f523f0e40de3456da628654b529402a4a83951def37f9da2d8a4b466b9b41
Jul 28 01:19:32.686143 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:32.686119    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"]
Jul 28 01:19:32.687782 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:19:32.687761    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73aa9bb6_bafd_4042_bc18_79d8573646f4.slice/crio-3177bd54f83a77ba191ed4b8c563842cfe4ad64517ee92bcb9358a8ca76fed50 WatchSource:0}: Error finding container 3177bd54f83a77ba191ed4b8c563842cfe4ad64517ee92bcb9358a8ca76fed50: Status 404 returned error can't find the container with id 3177bd54f83a77ba191ed4b8c563842cfe4ad64517ee92bcb9358a8ca76fed50
Jul 28 01:19:33.381753 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.381719    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4" event={"ID":"ca1e983d-d048-47d2-be21-339a4d32279c","Type":"ContainerStarted","Data":"ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a"}
Jul 28 01:19:33.381947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.381784    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:33.381947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.381808    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4" event={"ID":"ca1e983d-d048-47d2-be21-339a4d32279c","Type":"ContainerStarted","Data":"586f523f0e40de3456da628654b529402a4a83951def37f9da2d8a4b466b9b41"}
Jul 28 01:19:33.383288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.383252    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn" event={"ID":"73aa9bb6-bafd-4042-bc18-79d8573646f4","Type":"ContainerStarted","Data":"ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323"}
Jul 28 01:19:33.383399 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.383306    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:33.383399 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.383325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn" event={"ID":"73aa9bb6-bafd-4042-bc18-79d8573646f4","Type":"ContainerStarted","Data":"3177bd54f83a77ba191ed4b8c563842cfe4ad64517ee92bcb9358a8ca76fed50"}
Jul 28 01:19:33.397849 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.397807    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4" podStartSLOduration=1.39779384 podStartE2EDuration="1.39779384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:19:33.395889436 +0000 UTC m=+1978.145028161" watchObservedRunningTime="2026-07-28 01:19:33.39779384 +0000 UTC m=+1978.146932587"
Jul 28 01:19:33.412472 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:33.412426    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn" podStartSLOduration=1.4124130240000001 podStartE2EDuration="1.412413024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:19:33.41104904 +0000 UTC m=+1978.160187766" watchObservedRunningTime="2026-07-28 01:19:33.412413024 +0000 UTC m=+1978.161551747"
Jul 28 01:19:35.922759 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:35.922728    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:44.389734 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:44.389704    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:19:44.390188 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:44.390110    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:19:49.043613 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.043578    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"]
Jul 28 01:19:49.046838 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.046818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.051573 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.051549    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7ce60076\"/\"default-dockercfg-gf7ls\""
Jul 28 01:19:49.051724 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.051549    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7ce60076\"/\"openshift-service-ca.crt\""
Jul 28 01:19:49.051724 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.051555    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-7ce60076\"/\"e2e-lifecycle-model-7ce60076-kserve-self-signed-certs\""
Jul 28 01:19:49.051958 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.051934    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-7ce60076\"/\"kube-root-ca.crt\""
Jul 28 01:19:49.062576 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.062554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"]
Jul 28 01:19:49.086791 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.086767    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:19:49.182320 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.182292    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-model-cache\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.182479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.182329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hld2\" (UniqueName: \"kubernetes.io/projected/75659bd6-67ce-414a-b057-75f6ac7cc25b-kube-api-access-8hld2\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.182479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.182372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-tmp-dir\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.182479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.182410    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75659bd6-67ce-414a-b057-75f6ac7cc25b-tls-certs\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.182479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.182432    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-dshm\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.182479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.182453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-home\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.283810 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.283776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-tmp-dir\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.283951 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.283825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75659bd6-67ce-414a-b057-75f6ac7cc25b-tls-certs\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.283951 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.283857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-dshm\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.283951 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.283890    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-home\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.284117 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.283953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-model-cache\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.284117 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.283990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hld2\" (UniqueName: \"kubernetes.io/projected/75659bd6-67ce-414a-b057-75f6ac7cc25b-kube-api-access-8hld2\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.284264 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.284215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-tmp-dir\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.284359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.284327    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-home\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.284420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.284356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-model-cache\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.286092 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.286072    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-dshm\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.286349 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.286333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75659bd6-67ce-414a-b057-75f6ac7cc25b-tls-certs\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.300020 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.299970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hld2\" (UniqueName: \"kubernetes.io/projected/75659bd6-67ce-414a-b057-75f6ac7cc25b-kube-api-access-8hld2\") pod \"e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") " pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.356047 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.356012    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:49.488231 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:49.488191    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"]
Jul 28 01:19:49.489216 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:19:49.489188    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75659bd6_67ce_414a_b057_75f6ac7cc25b.slice/crio-e9c2bdcdb47b20473d27c28697227fcc665038900c945a988fb5d6744af03fb3 WatchSource:0}: Error finding container e9c2bdcdb47b20473d27c28697227fcc665038900c945a988fb5d6744af03fb3: Status 404 returned error can't find the container with id e9c2bdcdb47b20473d27c28697227fcc665038900c945a988fb5d6744af03fb3
Jul 28 01:19:50.465382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:50.465347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v" event={"ID":"75659bd6-67ce-414a-b057-75f6ac7cc25b","Type":"ContainerStarted","Data":"8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4"}
Jul 28 01:19:50.465382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:50.465387    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v" event={"ID":"75659bd6-67ce-414a-b057-75f6ac7cc25b","Type":"ContainerStarted","Data":"e9c2bdcdb47b20473d27c28697227fcc665038900c945a988fb5d6744af03fb3"}
Jul 28 01:19:50.484849 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:50.484805    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v" podStartSLOduration=1.484790572 podStartE2EDuration="1.484790572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:19:50.484066073 +0000 UTC m=+1995.233204798" watchObservedRunningTime="2026-07-28 01:19:50.484790572 +0000 UTC m=+1995.233929295"
Jul 28 01:19:59.356394 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:59.356363    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:59.356781 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:59.356439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:59.368882 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:59.368859    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:19:59.521893 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:19:59.521868    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:20:06.203561 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:06.203511    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:06.332520 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:06.332472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:26.045824 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.045753    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:26.090761 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.090723    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"]
Jul 28 01:20:26.091063 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.091024    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn" podUID="73aa9bb6-bafd-4042-bc18-79d8573646f4" containerName="payload-processing" containerID="cri-o://ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323" gracePeriod=30
Jul 28 01:20:26.122199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.122168    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"]
Jul 28 01:20:26.122467 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.122441    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4" podUID="ca1e983d-d048-47d2-be21-339a4d32279c" containerName="payload-pre-processing" containerID="cri-o://ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a" gracePeriod=30
Jul 28 01:20:26.164021 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.163905    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:26.262754 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.262692    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:26.395294 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.395273    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:20:26.398282 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.398264    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:20:26.517038 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.517000    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa9bb6-bafd-4042-bc18-79d8573646f4-plugins-config-volume\") pod \"73aa9bb6-bafd-4042-bc18-79d8573646f4\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") "
Jul 28 01:20:26.517207 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.517068    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9shn\" (UniqueName: \"kubernetes.io/projected/73aa9bb6-bafd-4042-bc18-79d8573646f4-kube-api-access-q9shn\") pod \"73aa9bb6-bafd-4042-bc18-79d8573646f4\" (UID: \"73aa9bb6-bafd-4042-bc18-79d8573646f4\") "
Jul 28 01:20:26.517207 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.517100    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1e983d-d048-47d2-be21-339a4d32279c-plugins-config-volume\") pod \"ca1e983d-d048-47d2-be21-339a4d32279c\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") "
Jul 28 01:20:26.517207 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.517145    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq2sv\" (UniqueName: \"kubernetes.io/projected/ca1e983d-d048-47d2-be21-339a4d32279c-kube-api-access-sq2sv\") pod \"ca1e983d-d048-47d2-be21-339a4d32279c\" (UID: \"ca1e983d-d048-47d2-be21-339a4d32279c\") "
Jul 28 01:20:26.517423 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.517397    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73aa9bb6-bafd-4042-bc18-79d8573646f4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73aa9bb6-bafd-4042-bc18-79d8573646f4" (UID: "73aa9bb6-bafd-4042-bc18-79d8573646f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:20:26.517515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.517496    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca1e983d-d048-47d2-be21-339a4d32279c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ca1e983d-d048-47d2-be21-339a4d32279c" (UID: "ca1e983d-d048-47d2-be21-339a4d32279c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:20:26.519203 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.519181    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73aa9bb6-bafd-4042-bc18-79d8573646f4-kube-api-access-q9shn" (OuterVolumeSpecName: "kube-api-access-q9shn") pod "73aa9bb6-bafd-4042-bc18-79d8573646f4" (UID: "73aa9bb6-bafd-4042-bc18-79d8573646f4"). InnerVolumeSpecName "kube-api-access-q9shn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:20:26.519300 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.519243    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca1e983d-d048-47d2-be21-339a4d32279c-kube-api-access-sq2sv" (OuterVolumeSpecName: "kube-api-access-sq2sv") pod "ca1e983d-d048-47d2-be21-339a4d32279c" (UID: "ca1e983d-d048-47d2-be21-339a4d32279c"). InnerVolumeSpecName "kube-api-access-sq2sv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:20:26.618703 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.618612    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aa9bb6-bafd-4042-bc18-79d8573646f4-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.618703 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.618646    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9shn\" (UniqueName: \"kubernetes.io/projected/73aa9bb6-bafd-4042-bc18-79d8573646f4-kube-api-access-q9shn\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.618703 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.618656    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca1e983d-d048-47d2-be21-339a4d32279c-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.618703 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.618665    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2sv\" (UniqueName: \"kubernetes.io/projected/ca1e983d-d048-47d2-be21-339a4d32279c-kube-api-access-sq2sv\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:26.637989 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.637958    2576 generic.go:356] "Generic (PLEG): container finished" podID="73aa9bb6-bafd-4042-bc18-79d8573646f4" containerID="ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323" exitCode=0
Jul 28 01:20:26.638153 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.637999    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"
Jul 28 01:20:26.638153 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.638043    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn" event={"ID":"73aa9bb6-bafd-4042-bc18-79d8573646f4","Type":"ContainerDied","Data":"ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323"}
Jul 28 01:20:26.638153 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.638089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn" event={"ID":"73aa9bb6-bafd-4042-bc18-79d8573646f4","Type":"ContainerDied","Data":"3177bd54f83a77ba191ed4b8c563842cfe4ad64517ee92bcb9358a8ca76fed50"}
Jul 28 01:20:26.638153 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.638110    2576 scope.go:117] "RemoveContainer" containerID="ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323"
Jul 28 01:20:26.639417 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.639398    2576 generic.go:356] "Generic (PLEG): container finished" podID="ca1e983d-d048-47d2-be21-339a4d32279c" containerID="ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a" exitCode=0
Jul 28 01:20:26.639502 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.639425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4" event={"ID":"ca1e983d-d048-47d2-be21-339a4d32279c","Type":"ContainerDied","Data":"ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a"}
Jul 28 01:20:26.639502 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.639475    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4" event={"ID":"ca1e983d-d048-47d2-be21-339a4d32279c","Type":"ContainerDied","Data":"586f523f0e40de3456da628654b529402a4a83951def37f9da2d8a4b466b9b41"}
Jul 28 01:20:26.639502 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.639435    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"
Jul 28 01:20:26.648184 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.648163    2576 scope.go:117] "RemoveContainer" containerID="ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323"
Jul 28 01:20:26.648579 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:20:26.648541    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323\": container with ID starting with ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323 not found: ID does not exist" containerID="ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323"
Jul 28 01:20:26.648654 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.648575    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323"} err="failed to get container status \"ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323\": rpc error: code = NotFound desc = could not find container \"ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323\": container with ID starting with ed935be7c7795568b4b6eddd90dcd06c7d363147293ef9af834b67d480bad323 not found: ID does not exist"
Jul 28 01:20:26.648654 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.648599    2576 scope.go:117] "RemoveContainer" containerID="ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a"
Jul 28 01:20:26.656714 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.656693    2576 scope.go:117] "RemoveContainer" containerID="ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a"
Jul 28 01:20:26.656953 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:20:26.656933    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a\": container with ID starting with ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a not found: ID does not exist" containerID="ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a"
Jul 28 01:20:26.657017 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.656958    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a"} err="failed to get container status \"ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a\": rpc error: code = NotFound desc = could not find container \"ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a\": container with ID starting with ac61223d04476416499dc75725a9407a1d4cb9c6d5aa8e7385af142f20da9e5a not found: ID does not exist"
Jul 28 01:20:26.665535 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.665514    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"]
Jul 28 01:20:26.671967 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.671946    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-7ce60076-67fb4d4464-wzsb4"]
Jul 28 01:20:26.682586 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.682566    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"]
Jul 28 01:20:26.688677 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:26.688651    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-7ce60076-786b5b5f4-622cn"]
Jul 28 01:20:27.751166 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:27.751128    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73aa9bb6-bafd-4042-bc18-79d8573646f4" path="/var/lib/kubelet/pods/73aa9bb6-bafd-4042-bc18-79d8573646f4/volumes"
Jul 28 01:20:27.751544 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:27.751514    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca1e983d-d048-47d2-be21-339a4d32279c" path="/var/lib/kubelet/pods/ca1e983d-d048-47d2-be21-339a4d32279c/volumes"
Jul 28 01:20:30.383980 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:30.383947    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:33.742615 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:33.742575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:38.705574 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:38.705540    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"]
Jul 28 01:20:38.831500 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:38.831472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:47.702264 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:47.702207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:51.511538 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:51.511492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:20:53.723960 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:53.723923    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v" podUID="75659bd6-67ce-414a-b057-75f6ac7cc25b" containerName="main" containerID="cri-o://8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4" gracePeriod=45
Jul 28 01:20:53.977563 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:53.977504    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:20:54.149420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149395    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75659bd6-67ce-414a-b057-75f6ac7cc25b-tls-certs\") pod \"75659bd6-67ce-414a-b057-75f6ac7cc25b\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") "
Jul 28 01:20:54.149587 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149435    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-tmp-dir\") pod \"75659bd6-67ce-414a-b057-75f6ac7cc25b\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") "
Jul 28 01:20:54.149587 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149466    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-dshm\") pod \"75659bd6-67ce-414a-b057-75f6ac7cc25b\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") "
Jul 28 01:20:54.149587 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-model-cache\") pod \"75659bd6-67ce-414a-b057-75f6ac7cc25b\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") "
Jul 28 01:20:54.149587 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149568    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hld2\" (UniqueName: \"kubernetes.io/projected/75659bd6-67ce-414a-b057-75f6ac7cc25b-kube-api-access-8hld2\") pod \"75659bd6-67ce-414a-b057-75f6ac7cc25b\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") "
Jul 28 01:20:54.149774 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149607    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-home\") pod \"75659bd6-67ce-414a-b057-75f6ac7cc25b\" (UID: \"75659bd6-67ce-414a-b057-75f6ac7cc25b\") "
Jul 28 01:20:54.149774 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149713    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "75659bd6-67ce-414a-b057-75f6ac7cc25b" (UID: "75659bd6-67ce-414a-b057-75f6ac7cc25b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:20:54.149886 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149774    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-model-cache" (OuterVolumeSpecName: "model-cache") pod "75659bd6-67ce-414a-b057-75f6ac7cc25b" (UID: "75659bd6-67ce-414a-b057-75f6ac7cc25b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:20:54.149944 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149909    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:54.149944 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149923    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:54.149944 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.149928    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-home" (OuterVolumeSpecName: "home") pod "75659bd6-67ce-414a-b057-75f6ac7cc25b" (UID: "75659bd6-67ce-414a-b057-75f6ac7cc25b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:20:54.151671 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.151644    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-dshm" (OuterVolumeSpecName: "dshm") pod "75659bd6-67ce-414a-b057-75f6ac7cc25b" (UID: "75659bd6-67ce-414a-b057-75f6ac7cc25b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:20:54.151773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.151688    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75659bd6-67ce-414a-b057-75f6ac7cc25b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "75659bd6-67ce-414a-b057-75f6ac7cc25b" (UID: "75659bd6-67ce-414a-b057-75f6ac7cc25b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:20:54.151773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.151709    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75659bd6-67ce-414a-b057-75f6ac7cc25b-kube-api-access-8hld2" (OuterVolumeSpecName: "kube-api-access-8hld2") pod "75659bd6-67ce-414a-b057-75f6ac7cc25b" (UID: "75659bd6-67ce-414a-b057-75f6ac7cc25b"). InnerVolumeSpecName "kube-api-access-8hld2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:20:54.250835 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.250775    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hld2\" (UniqueName: \"kubernetes.io/projected/75659bd6-67ce-414a-b057-75f6ac7cc25b-kube-api-access-8hld2\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:54.250835 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.250798    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:54.250835 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.250807    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75659bd6-67ce-414a-b057-75f6ac7cc25b-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:54.250835 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.250815    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75659bd6-67ce-414a-b057-75f6ac7cc25b-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:20:54.766992 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.766961    2576 generic.go:356] "Generic (PLEG): container finished" podID="75659bd6-67ce-414a-b057-75f6ac7cc25b" containerID="8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4" exitCode=0
Jul 28 01:20:54.767404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.767003    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"
Jul 28 01:20:54.767404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.767028    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v" event={"ID":"75659bd6-67ce-414a-b057-75f6ac7cc25b","Type":"ContainerDied","Data":"8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4"}
Jul 28 01:20:54.767404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.767059    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v" event={"ID":"75659bd6-67ce-414a-b057-75f6ac7cc25b","Type":"ContainerDied","Data":"e9c2bdcdb47b20473d27c28697227fcc665038900c945a988fb5d6744af03fb3"}
Jul 28 01:20:54.767404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.767076    2576 scope.go:117] "RemoveContainer" containerID="8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4"
Jul 28 01:20:54.775758 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.775742    2576 scope.go:117] "RemoveContainer" containerID="8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4"
Jul 28 01:20:54.776000 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:20:54.775978    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4\": container with ID starting with 8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4 not found: ID does not exist" containerID="8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4"
Jul 28 01:20:54.776054 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.776008    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4"} err="failed to get container status \"8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4\": rpc error: code = NotFound desc = could not find container \"8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4\": container with ID starting with 8f47140c1acd570b774038bb7179ab8f4479399c1ced18c4264a453a86f29df4 not found: ID does not exist"
Jul 28 01:20:54.798297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.798270    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"]
Jul 28 01:20:54.804186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:54.804167    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-7ce60076/e2e-lifecycle-model-7ce60076-kserve-6df6b7b67f-kgt7v"]
Jul 28 01:20:55.750583 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:20:55.750552    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75659bd6-67ce-414a-b057-75f6ac7cc25b" path="/var/lib/kubelet/pods/75659bd6-67ce-414a-b057-75f6ac7cc25b/volumes"
Jul 28 01:21:15.971113 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:15.971023    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:16.161724 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:16.161653    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:22.928141 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:22.928106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:28.615667 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:28.615628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:28.705661 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:28.705629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:28.969539 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:28.969451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:29.191533 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:29.191498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:30.443666 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:30.443632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:30.660098 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:30.660070    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:31.361505 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:31.361466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:31.522429 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:31.522383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:31.615155 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:31.615068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:32.911134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:32.911102    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:38.807407 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:38.807367    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:38.907449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:38.907411    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:39.482682 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:39.482649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:45.684082 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:45.684048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:45.806078 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:45.806047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:45.900096 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:45.900059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:56.385824 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:56.385786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:21:56.699105 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:21:56.699027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:17.684760 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:17.684716    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:23.929367 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:23.929321    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:24.033798 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:24.033765    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:24.277175 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:24.277102    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:24.620593 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:24.620562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:24.697740 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:24.697710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:27.442522 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:27.442483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:27.549393 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:27.549350    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:27.638147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:27.638109    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:27.839034 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:27.839002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:28.379893 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.379857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:28.476312 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.476281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:28.514802 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.514773    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"]
Jul 28 01:22:28.515104 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515091    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca1e983d-d048-47d2-be21-339a4d32279c" containerName="payload-pre-processing"
Jul 28 01:22:28.515147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515105    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca1e983d-d048-47d2-be21-339a4d32279c" containerName="payload-pre-processing"
Jul 28 01:22:28.515147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515130    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73aa9bb6-bafd-4042-bc18-79d8573646f4" containerName="payload-processing"
Jul 28 01:22:28.515147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515135    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="73aa9bb6-bafd-4042-bc18-79d8573646f4" containerName="payload-processing"
Jul 28 01:22:28.515147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515142    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75659bd6-67ce-414a-b057-75f6ac7cc25b" containerName="main"
Jul 28 01:22:28.515147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515147    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="75659bd6-67ce-414a-b057-75f6ac7cc25b" containerName="main"
Jul 28 01:22:28.515325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515192    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="73aa9bb6-bafd-4042-bc18-79d8573646f4" containerName="payload-processing"
Jul 28 01:22:28.515325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515200    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="75659bd6-67ce-414a-b057-75f6ac7cc25b" containerName="main"
Jul 28 01:22:28.515325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.515209    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca1e983d-d048-47d2-be21-339a4d32279c" containerName="payload-pre-processing"
Jul 28 01:22:28.518067 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.518047    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.520616 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.520598    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ee92ea\""
Jul 28 01:22:28.531819 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.531800    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"]
Jul 28 01:22:28.590293 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.590264    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"]
Jul 28 01:22:28.593473 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.593458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.596776 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.596754    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ee92ea-dockercfg-x4456\""
Jul 28 01:22:28.607957 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.607921    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"]
Jul 28 01:22:28.634200 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.634129    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl9ln\" (UniqueName: \"kubernetes.io/projected/7537664e-242e-45ef-b2c6-0c88eba7d697-kube-api-access-vl9ln\") pod \"payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.634346 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.634279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7537664e-242e-45ef-b2c6-0c88eba7d697-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.734990 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.734964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7537664e-242e-45ef-b2c6-0c88eba7d697-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.735101 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.735004    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c95fda-7a44-4744-9d80-23d171bb4cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.735101 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.735035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl9ln\" (UniqueName: \"kubernetes.io/projected/7537664e-242e-45ef-b2c6-0c88eba7d697-kube-api-access-vl9ln\") pod \"payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.735101 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.735084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79m5w\" (UniqueName: \"kubernetes.io/projected/d5c95fda-7a44-4744-9d80-23d171bb4cee-kube-api-access-79m5w\") pod \"payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.735683 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.735662    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7537664e-242e-45ef-b2c6-0c88eba7d697-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.744696 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.744672    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl9ln\" (UniqueName: \"kubernetes.io/projected/7537664e-242e-45ef-b2c6-0c88eba7d697-kube-api-access-vl9ln\") pod \"payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.827511 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.827485    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:28.835566 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.835540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c95fda-7a44-4744-9d80-23d171bb4cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.835670 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.835584    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79m5w\" (UniqueName: \"kubernetes.io/projected/d5c95fda-7a44-4744-9d80-23d171bb4cee-kube-api-access-79m5w\") pod \"payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.836376 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.836355    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c95fda-7a44-4744-9d80-23d171bb4cee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.852112 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.852088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79m5w\" (UniqueName: \"kubernetes.io/projected/d5c95fda-7a44-4744-9d80-23d171bb4cee-kube-api-access-79m5w\") pod \"payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.903437 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.903372    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:28.966707 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:28.966682    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"]
Jul 28 01:22:28.968571 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:22:28.968343    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7537664e_242e_45ef_b2c6_0c88eba7d697.slice/crio-3f2224e32a8b531b5da956be2f14b7b9d43531865ccd03b6c1b0efb06ea39a22 WatchSource:0}: Error finding container 3f2224e32a8b531b5da956be2f14b7b9d43531865ccd03b6c1b0efb06ea39a22: Status 404 returned error can't find the container with id 3f2224e32a8b531b5da956be2f14b7b9d43531865ccd03b6c1b0efb06ea39a22
Jul 28 01:22:29.056208 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.056181    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"]
Jul 28 01:22:29.057613 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:22:29.057588    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5c95fda_7a44_4744_9d80_23d171bb4cee.slice/crio-a8a03a30e4fca848fc31f440d610a6bbf31f1eb924e957a9cbad68602da8a15f WatchSource:0}: Error finding container a8a03a30e4fca848fc31f440d610a6bbf31f1eb924e957a9cbad68602da8a15f: Status 404 returned error can't find the container with id a8a03a30e4fca848fc31f440d610a6bbf31f1eb924e957a9cbad68602da8a15f
Jul 28 01:22:29.186448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.186344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8" event={"ID":"d5c95fda-7a44-4744-9d80-23d171bb4cee","Type":"ContainerStarted","Data":"1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0"}
Jul 28 01:22:29.186609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.186451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8" event={"ID":"d5c95fda-7a44-4744-9d80-23d171bb4cee","Type":"ContainerStarted","Data":"a8a03a30e4fca848fc31f440d610a6bbf31f1eb924e957a9cbad68602da8a15f"}
Jul 28 01:22:29.186609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.186484    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:29.188137 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.188104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh" event={"ID":"7537664e-242e-45ef-b2c6-0c88eba7d697","Type":"ContainerStarted","Data":"c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172"}
Jul 28 01:22:29.188304 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.188152    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:29.188304 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.188174    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh" event={"ID":"7537664e-242e-45ef-b2c6-0c88eba7d697","Type":"ContainerStarted","Data":"3f2224e32a8b531b5da956be2f14b7b9d43531865ccd03b6c1b0efb06ea39a22"}
Jul 28 01:22:29.206714 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:29.206673    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8" podStartSLOduration=1.206659827 podStartE2EDuration="1.206659827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:29.204619762 +0000 UTC m=+2153.953758487" watchObservedRunningTime="2026-07-28 01:22:29.206659827 +0000 UTC m=+2153.955798551"
Jul 28 01:22:31.929714 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:31.929681    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:32.267195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:32.267119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:32.621276 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:32.621239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:32.906001 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:32.905922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:35.790963 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:35.790917    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:35.892886 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:35.892856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:36.009378 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.009335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:36.464540 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.464505    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:36.581278 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.581245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:36.629195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.629150    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh" podStartSLOduration=8.629133245 podStartE2EDuration="8.629133245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:29.23544368 +0000 UTC m=+2153.984582405" watchObservedRunningTime="2026-07-28 01:22:36.629133245 +0000 UTC m=+2161.378271978"
Jul 28 01:22:36.630148 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.630128    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"]
Jul 28 01:22:36.633698 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.633683    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.636862 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.636840    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-f1343b\""
Jul 28 01:22:36.666343 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.666319    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"]
Jul 28 01:22:36.689020 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.688994    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"]
Jul 28 01:22:36.692256 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.692217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.695868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.695850    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-f1343b-dockercfg-vvlfp\""
Jul 28 01:22:36.704704 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.704681    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fa6e7c-2302-42c0-a379-541c4381f11d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.704810 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.704728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf475\" (UniqueName: \"kubernetes.io/projected/40fa6e7c-2302-42c0-a379-541c4381f11d-kube-api-access-mf475\") pod \"payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.718852 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.718796    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"]
Jul 28 01:22:36.784168 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.784148    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:36.805344 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.805321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74zpv\" (UniqueName: \"kubernetes.io/projected/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-kube-api-access-74zpv\") pod \"payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.805618 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.805352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.805618 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.805377    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fa6e7c-2302-42c0-a379-541c4381f11d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.805618 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.805431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf475\" (UniqueName: \"kubernetes.io/projected/40fa6e7c-2302-42c0-a379-541c4381f11d-kube-api-access-mf475\") pod \"payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.805959 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.805943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fa6e7c-2302-42c0-a379-541c4381f11d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.814472 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.814452    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf475\" (UniqueName: \"kubernetes.io/projected/40fa6e7c-2302-42c0-a379-541c4381f11d-kube-api-access-mf475\") pod \"payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:36.906301 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.906274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74zpv\" (UniqueName: \"kubernetes.io/projected/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-kube-api-access-74zpv\") pod \"payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.906435 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.906306    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.906952 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.906934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.914404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.914372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74zpv\" (UniqueName: \"kubernetes.io/projected/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-kube-api-access-74zpv\") pod \"payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:36.944340 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:36.944315    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:37.002690 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.002636    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:37.083849 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.083814    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"]
Jul 28 01:22:37.085209 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:22:37.085183    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40fa6e7c_2302_42c0_a379_541c4381f11d.slice/crio-946d6cdfffd24490283c5743c9b57b96c5ef1b0909a756372a70fe0b88b78d55 WatchSource:0}: Error finding container 946d6cdfffd24490283c5743c9b57b96c5ef1b0909a756372a70fe0b88b78d55: Status 404 returned error can't find the container with id 946d6cdfffd24490283c5743c9b57b96c5ef1b0909a756372a70fe0b88b78d55
Jul 28 01:22:37.141514 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.141491    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"]
Jul 28 01:22:37.143428 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:22:37.143402    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11d995bb_b1a3_4c8a_8cd4_e093f8edc643.slice/crio-40a706ad73f52e6d03d3676529e6b359374c85e57ee1a45c9800a8051037f115 WatchSource:0}: Error finding container 40a706ad73f52e6d03d3676529e6b359374c85e57ee1a45c9800a8051037f115: Status 404 returned error can't find the container with id 40a706ad73f52e6d03d3676529e6b359374c85e57ee1a45c9800a8051037f115
Jul 28 01:22:37.223528 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.223496    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv" event={"ID":"40fa6e7c-2302-42c0-a379-541c4381f11d","Type":"ContainerStarted","Data":"dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07"}
Jul 28 01:22:37.223658 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.223554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv" event={"ID":"40fa6e7c-2302-42c0-a379-541c4381f11d","Type":"ContainerStarted","Data":"946d6cdfffd24490283c5743c9b57b96c5ef1b0909a756372a70fe0b88b78d55"}
Jul 28 01:22:37.223730 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.223704    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:37.225116 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.225094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78" event={"ID":"11d995bb-b1a3-4c8a-8cd4-e093f8edc643","Type":"ContainerStarted","Data":"19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe"}
Jul 28 01:22:37.225203 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.225133    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78" event={"ID":"11d995bb-b1a3-4c8a-8cd4-e093f8edc643","Type":"ContainerStarted","Data":"40a706ad73f52e6d03d3676529e6b359374c85e57ee1a45c9800a8051037f115"}
Jul 28 01:22:37.225203 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.225155    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:37.243261 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.243146    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv" podStartSLOduration=1.243130396 podStartE2EDuration="1.243130396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:37.24254003 +0000 UTC m=+2161.991678754" watchObservedRunningTime="2026-07-28 01:22:37.243130396 +0000 UTC m=+2161.992269121"
Jul 28 01:22:37.259620 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:37.259581    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78" podStartSLOduration=1.259567345 podStartE2EDuration="1.259567345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:22:37.258734705 +0000 UTC m=+2162.007873430" watchObservedRunningTime="2026-07-28 01:22:37.259567345 +0000 UTC m=+2162.008706069"
Jul 28 01:22:40.194395 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:40.194363    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:22:40.194773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:40.194434    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:22:40.288632 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:40.288602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:22:48.231907 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:48.231875    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:22:48.232353 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:48.231958    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:22:59.995866 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:59.995835    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"]
Jul 28 01:22:59.999558 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:22:59.999536    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.003723 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.003692    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ee92ea\"/\"openshift-service-ca.crt\""
Jul 28 01:23:00.003846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.003721    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ee92ea\"/\"default-dockercfg-ck4bs\""
Jul 28 01:23:00.003846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.003729    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-ee92ea\"/\"test-model-ee92ea-kserve-self-signed-certs\""
Jul 28 01:23:00.003846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.003734    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-ee92ea\"/\"kube-root-ca.crt\""
Jul 28 01:23:00.009131 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.009104    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"]
Jul 28 01:23:00.033275 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.033232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:23:00.094634 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.094601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-home\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.094634 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.094634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-dshm\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.094823 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.094660    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-model-cache\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.094823 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.094685    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f750fa-4415-45a3-a4a4-ff4af601404d-tls-certs\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.094823 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.094719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-tmp-dir\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.094823 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.094749    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrffr\" (UniqueName: \"kubernetes.io/projected/a6f750fa-4415-45a3-a4a4-ff4af601404d-kube-api-access-jrffr\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195282 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195245    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-home\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-dshm\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195327    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-model-cache\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f750fa-4415-45a3-a4a4-ff4af601404d-tls-certs\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195374    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-tmp-dir\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrffr\" (UniqueName: \"kubernetes.io/projected/a6f750fa-4415-45a3-a4a4-ff4af601404d-kube-api-access-jrffr\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195735 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195598    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-home\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195791 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-model-cache\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.195842 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.195813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-tmp-dir\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.197578 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.197556    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-dshm\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.197909 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.197891    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f750fa-4415-45a3-a4a4-ff4af601404d-tls-certs\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.203525 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.203505    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrffr\" (UniqueName: \"kubernetes.io/projected/a6f750fa-4415-45a3-a4a4-ff4af601404d-kube-api-access-jrffr\") pod \"test-model-ee92ea-kserve-866f87547-9lgcq\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") " pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.310785 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.310766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:00.435195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:00.435170    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"]
Jul 28 01:23:00.437111 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:23:00.437081    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6f750fa_4415_45a3_a4a4_ff4af601404d.slice/crio-ed4ed4a8d3df5ab166ddca4f516aa404c317369c67a4405eb5ffb141b5024e88 WatchSource:0}: Error finding container ed4ed4a8d3df5ab166ddca4f516aa404c317369c67a4405eb5ffb141b5024e88: Status 404 returned error can't find the container with id ed4ed4a8d3df5ab166ddca4f516aa404c317369c67a4405eb5ffb141b5024e88
Jul 28 01:23:01.335328 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:01.335294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq" event={"ID":"a6f750fa-4415-45a3-a4a4-ff4af601404d","Type":"ContainerStarted","Data":"83e0ffa4f739add6ce5c26ae89ae53bfdf4b5a8e9b0c8db6877976da5f9d0388"}
Jul 28 01:23:01.335328 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:01.335338    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq" event={"ID":"a6f750fa-4415-45a3-a4a4-ff4af601404d","Type":"ContainerStarted","Data":"ed4ed4a8d3df5ab166ddca4f516aa404c317369c67a4405eb5ffb141b5024e88"}
Jul 28 01:23:01.352142 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:01.352093    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq" podStartSLOduration=2.352076213 podStartE2EDuration="2.352076213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:23:01.350912081 +0000 UTC m=+2186.100050798" watchObservedRunningTime="2026-07-28 01:23:01.352076213 +0000 UTC m=+2186.101214936"
Jul 28 01:23:10.311486 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:10.311452    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:10.311486 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:10.311492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:10.323877 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:10.323850    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:10.388704 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:10.388676    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:23:49.972606 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:23:49.972568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:05.495072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:05.494988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:05.595585 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:05.595556    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:14.463240 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:14.463178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:40.675980 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:40.675943    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"]
Jul 28 01:24:40.699271 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:40.699244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:41.575505 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:41.575472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:52.226710 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.226674    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:52.246518 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.246482    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"]
Jul 28 01:24:52.246825 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.246793    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh" podUID="7537664e-242e-45ef-b2c6-0c88eba7d697" containerName="payload-processing" containerID="cri-o://c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172" gracePeriod=30
Jul 28 01:24:52.253779 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.253738    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"]
Jul 28 01:24:52.254006 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.253978    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8" podUID="d5c95fda-7a44-4744-9d80-23d171bb4cee" containerName="payload-pre-processing" containerID="cri-o://1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0" gracePeriod=30
Jul 28 01:24:52.326756 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.326729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:52.388477 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.388444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:52.520508 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.520479    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:24:52.523482 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.523464    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:24:52.540729 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.540704    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7537664e-242e-45ef-b2c6-0c88eba7d697-plugins-config-volume\") pod \"7537664e-242e-45ef-b2c6-0c88eba7d697\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") "
Jul 28 01:24:52.540828 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.540746    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79m5w\" (UniqueName: \"kubernetes.io/projected/d5c95fda-7a44-4744-9d80-23d171bb4cee-kube-api-access-79m5w\") pod \"d5c95fda-7a44-4744-9d80-23d171bb4cee\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") "
Jul 28 01:24:52.540828 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.540814    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c95fda-7a44-4744-9d80-23d171bb4cee-plugins-config-volume\") pod \"d5c95fda-7a44-4744-9d80-23d171bb4cee\" (UID: \"d5c95fda-7a44-4744-9d80-23d171bb4cee\") "
Jul 28 01:24:52.540920 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.540873    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl9ln\" (UniqueName: \"kubernetes.io/projected/7537664e-242e-45ef-b2c6-0c88eba7d697-kube-api-access-vl9ln\") pod \"7537664e-242e-45ef-b2c6-0c88eba7d697\" (UID: \"7537664e-242e-45ef-b2c6-0c88eba7d697\") "
Jul 28 01:24:52.541211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.541176    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7537664e-242e-45ef-b2c6-0c88eba7d697-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7537664e-242e-45ef-b2c6-0c88eba7d697" (UID: "7537664e-242e-45ef-b2c6-0c88eba7d697"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:24:52.541351 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.541201    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c95fda-7a44-4744-9d80-23d171bb4cee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5c95fda-7a44-4744-9d80-23d171bb4cee" (UID: "d5c95fda-7a44-4744-9d80-23d171bb4cee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:24:52.543352 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.543323    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7537664e-242e-45ef-b2c6-0c88eba7d697-kube-api-access-vl9ln" (OuterVolumeSpecName: "kube-api-access-vl9ln") pod "7537664e-242e-45ef-b2c6-0c88eba7d697" (UID: "7537664e-242e-45ef-b2c6-0c88eba7d697"). InnerVolumeSpecName "kube-api-access-vl9ln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:24:52.543622 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.543506    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c95fda-7a44-4744-9d80-23d171bb4cee-kube-api-access-79m5w" (OuterVolumeSpecName: "kube-api-access-79m5w") pod "d5c95fda-7a44-4744-9d80-23d171bb4cee" (UID: "d5c95fda-7a44-4744-9d80-23d171bb4cee"). InnerVolumeSpecName "kube-api-access-79m5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:24:52.642074 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.642046    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl9ln\" (UniqueName: \"kubernetes.io/projected/7537664e-242e-45ef-b2c6-0c88eba7d697-kube-api-access-vl9ln\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:52.642074 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.642070    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7537664e-242e-45ef-b2c6-0c88eba7d697-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:52.642074 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.642080    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79m5w\" (UniqueName: \"kubernetes.io/projected/d5c95fda-7a44-4744-9d80-23d171bb4cee-kube-api-access-79m5w\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:52.642384 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.642088    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5c95fda-7a44-4744-9d80-23d171bb4cee-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:52.838269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.838212    2576 generic.go:356] "Generic (PLEG): container finished" podID="d5c95fda-7a44-4744-9d80-23d171bb4cee" containerID="1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0" exitCode=0
Jul 28 01:24:52.838461 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.838273    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"
Jul 28 01:24:52.838461 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.838296    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8" event={"ID":"d5c95fda-7a44-4744-9d80-23d171bb4cee","Type":"ContainerDied","Data":"1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0"}
Jul 28 01:24:52.838461 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.838350    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8" event={"ID":"d5c95fda-7a44-4744-9d80-23d171bb4cee","Type":"ContainerDied","Data":"a8a03a30e4fca848fc31f440d610a6bbf31f1eb924e957a9cbad68602da8a15f"}
Jul 28 01:24:52.838461 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.838377    2576 scope.go:117] "RemoveContainer" containerID="1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0"
Jul 28 01:24:52.839644 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.839621    2576 generic.go:356] "Generic (PLEG): container finished" podID="7537664e-242e-45ef-b2c6-0c88eba7d697" containerID="c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172" exitCode=0
Jul 28 01:24:52.839725 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.839665    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"
Jul 28 01:24:52.839725 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.839690    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh" event={"ID":"7537664e-242e-45ef-b2c6-0c88eba7d697","Type":"ContainerDied","Data":"c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172"}
Jul 28 01:24:52.839845 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.839737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh" event={"ID":"7537664e-242e-45ef-b2c6-0c88eba7d697","Type":"ContainerDied","Data":"3f2224e32a8b531b5da956be2f14b7b9d43531865ccd03b6c1b0efb06ea39a22"}
Jul 28 01:24:52.847521 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.847507    2576 scope.go:117] "RemoveContainer" containerID="1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0"
Jul 28 01:24:52.847811 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:24:52.847776    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0\": container with ID starting with 1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0 not found: ID does not exist" containerID="1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0"
Jul 28 01:24:52.847891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.847806    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0"} err="failed to get container status \"1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0\": rpc error: code = NotFound desc = could not find container \"1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0\": container with ID starting with 1a526b0d2e3d3be52ddc60d237b34921db0cfaadbe2a595634340e905496e6c0 not found: ID does not exist"
Jul 28 01:24:52.847891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.847822    2576 scope.go:117] "RemoveContainer" containerID="c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172"
Jul 28 01:24:52.858586 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.858564    2576 scope.go:117] "RemoveContainer" containerID="c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172"
Jul 28 01:24:52.858938 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:24:52.858921    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172\": container with ID starting with c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172 not found: ID does not exist" containerID="c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172"
Jul 28 01:24:52.859006 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.858944    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172"} err="failed to get container status \"c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172\": rpc error: code = NotFound desc = could not find container \"c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172\": container with ID starting with c09c2dc677904c319535a0cdeb950219ca32d3292544718528fa87d0f015e172 not found: ID does not exist"
Jul 28 01:24:52.865134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.865108    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"]
Jul 28 01:24:52.867330 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.867308    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ee92ea-5b5bcfbd96-hndnh"]
Jul 28 01:24:52.878025 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.877997    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"]
Jul 28 01:24:52.884005 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:52.883982    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ee92ea-58c65dbcbd-plnm8"]
Jul 28 01:24:53.752545 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:53.752511    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7537664e-242e-45ef-b2c6-0c88eba7d697" path="/var/lib/kubelet/pods/7537664e-242e-45ef-b2c6-0c88eba7d697/volumes"
Jul 28 01:24:53.753006 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:53.752987    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5c95fda-7a44-4744-9d80-23d171bb4cee" path="/var/lib/kubelet/pods/d5c95fda-7a44-4744-9d80-23d171bb4cee/volumes"
Jul 28 01:24:55.694630 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.694593    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq" podUID="a6f750fa-4415-45a3-a4a4-ff4af601404d" containerName="main" containerID="cri-o://83e0ffa4f739add6ce5c26ae89ae53bfdf4b5a8e9b0c8db6877976da5f9d0388" gracePeriod=45
Jul 28 01:24:55.857147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.857117    2576 generic.go:356] "Generic (PLEG): container finished" podID="a6f750fa-4415-45a3-a4a4-ff4af601404d" containerID="83e0ffa4f739add6ce5c26ae89ae53bfdf4b5a8e9b0c8db6877976da5f9d0388" exitCode=0
Jul 28 01:24:55.857319 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.857187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq" event={"ID":"a6f750fa-4415-45a3-a4a4-ff4af601404d","Type":"ContainerDied","Data":"83e0ffa4f739add6ce5c26ae89ae53bfdf4b5a8e9b0c8db6877976da5f9d0388"}
Jul 28 01:24:55.950625 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.950567    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:24:55.970327 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970298    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-tmp-dir\") pod \"a6f750fa-4415-45a3-a4a4-ff4af601404d\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") "
Jul 28 01:24:55.970578 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970430    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-dshm\") pod \"a6f750fa-4415-45a3-a4a4-ff4af601404d\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") "
Jul 28 01:24:55.970578 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970496    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-home\") pod \"a6f750fa-4415-45a3-a4a4-ff4af601404d\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") "
Jul 28 01:24:55.970578 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970569    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f750fa-4415-45a3-a4a4-ff4af601404d-tls-certs\") pod \"a6f750fa-4415-45a3-a4a4-ff4af601404d\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") "
Jul 28 01:24:55.970887 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970605    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-model-cache\") pod \"a6f750fa-4415-45a3-a4a4-ff4af601404d\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") "
Jul 28 01:24:55.970887 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970651    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6f750fa-4415-45a3-a4a4-ff4af601404d" (UID: "a6f750fa-4415-45a3-a4a4-ff4af601404d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:24:55.970887 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970710    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrffr\" (UniqueName: \"kubernetes.io/projected/a6f750fa-4415-45a3-a4a4-ff4af601404d-kube-api-access-jrffr\") pod \"a6f750fa-4415-45a3-a4a4-ff4af601404d\" (UID: \"a6f750fa-4415-45a3-a4a4-ff4af601404d\") "
Jul 28 01:24:55.971043 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.970884    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-home" (OuterVolumeSpecName: "home") pod "a6f750fa-4415-45a3-a4a4-ff4af601404d" (UID: "a6f750fa-4415-45a3-a4a4-ff4af601404d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:24:55.971043 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.971001    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:55.971043 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.971021    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:55.971211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.971185    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6f750fa-4415-45a3-a4a4-ff4af601404d" (UID: "a6f750fa-4415-45a3-a4a4-ff4af601404d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:24:55.973840 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.973813    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-dshm" (OuterVolumeSpecName: "dshm") pod "a6f750fa-4415-45a3-a4a4-ff4af601404d" (UID: "a6f750fa-4415-45a3-a4a4-ff4af601404d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:24:55.974302 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.974276    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6f750fa-4415-45a3-a4a4-ff4af601404d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6f750fa-4415-45a3-a4a4-ff4af601404d" (UID: "a6f750fa-4415-45a3-a4a4-ff4af601404d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:24:55.974577 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:55.974554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f750fa-4415-45a3-a4a4-ff4af601404d-kube-api-access-jrffr" (OuterVolumeSpecName: "kube-api-access-jrffr") pod "a6f750fa-4415-45a3-a4a4-ff4af601404d" (UID: "a6f750fa-4415-45a3-a4a4-ff4af601404d"). InnerVolumeSpecName "kube-api-access-jrffr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:24:56.072076 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.072039    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:56.072076 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.072070    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6f750fa-4415-45a3-a4a4-ff4af601404d-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:56.072076 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.072081    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6f750fa-4415-45a3-a4a4-ff4af601404d-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:56.072327 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.072090    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrffr\" (UniqueName: \"kubernetes.io/projected/a6f750fa-4415-45a3-a4a4-ff4af601404d-kube-api-access-jrffr\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:24:56.779016 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.778985    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:56.863021 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.862983    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq" event={"ID":"a6f750fa-4415-45a3-a4a4-ff4af601404d","Type":"ContainerDied","Data":"ed4ed4a8d3df5ab166ddca4f516aa404c317369c67a4405eb5ffb141b5024e88"}
Jul 28 01:24:56.863021 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.863022    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"
Jul 28 01:24:56.863255 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.863041    2576 scope.go:117] "RemoveContainer" containerID="83e0ffa4f739add6ce5c26ae89ae53bfdf4b5a8e9b0c8db6877976da5f9d0388"
Jul 28 01:24:56.884867 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.884839    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"]
Jul 28 01:24:56.890905 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:56.890884    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-ee92ea/test-model-ee92ea-kserve-866f87547-9lgcq"]
Jul 28 01:24:57.190099 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:57.190034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:24:57.751741 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:24:57.751703    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6f750fa-4415-45a3-a4a4-ff4af601404d" path="/var/lib/kubelet/pods/a6f750fa-4415-45a3-a4a4-ff4af601404d/volumes"
Jul 28 01:25:03.103515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:03.103483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:03.203539 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:03.203507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:03.359715 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:03.359613    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:03.581301 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:03.581258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:14.216591 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.216552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:14.246569 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.246539    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"]
Jul 28 01:25:14.247023 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.246802    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv" podUID="40fa6e7c-2302-42c0-a379-541c4381f11d" containerName="payload-processing" containerID="cri-o://dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07" gracePeriod=30
Jul 28 01:25:14.255167 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.255139    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"]
Jul 28 01:25:14.255400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.255369    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78" podUID="11d995bb-b1a3-4c8a-8cd4-e093f8edc643" containerName="payload-pre-processing" containerID="cri-o://19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe" gracePeriod=30
Jul 28 01:25:14.289807 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.289778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:14.388239 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.388191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:14.531647 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.531625    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:25:14.534482 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.534463    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:25:14.635085 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.635060    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74zpv\" (UniqueName: \"kubernetes.io/projected/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-kube-api-access-74zpv\") pod \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") "
Jul 28 01:25:14.635248 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.635094    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fa6e7c-2302-42c0-a379-541c4381f11d-plugins-config-volume\") pod \"40fa6e7c-2302-42c0-a379-541c4381f11d\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") "
Jul 28 01:25:14.635248 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.635125    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-plugins-config-volume\") pod \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\" (UID: \"11d995bb-b1a3-4c8a-8cd4-e093f8edc643\") "
Jul 28 01:25:14.635248 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.635152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf475\" (UniqueName: \"kubernetes.io/projected/40fa6e7c-2302-42c0-a379-541c4381f11d-kube-api-access-mf475\") pod \"40fa6e7c-2302-42c0-a379-541c4381f11d\" (UID: \"40fa6e7c-2302-42c0-a379-541c4381f11d\") "
Jul 28 01:25:14.635528 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.635509    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11d995bb-b1a3-4c8a-8cd4-e093f8edc643" (UID: "11d995bb-b1a3-4c8a-8cd4-e093f8edc643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:25:14.635585 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.635514    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40fa6e7c-2302-42c0-a379-541c4381f11d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "40fa6e7c-2302-42c0-a379-541c4381f11d" (UID: "40fa6e7c-2302-42c0-a379-541c4381f11d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:25:14.637173 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.637156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-kube-api-access-74zpv" (OuterVolumeSpecName: "kube-api-access-74zpv") pod "11d995bb-b1a3-4c8a-8cd4-e093f8edc643" (UID: "11d995bb-b1a3-4c8a-8cd4-e093f8edc643"). InnerVolumeSpecName "kube-api-access-74zpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:25:14.637266 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.637215    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40fa6e7c-2302-42c0-a379-541c4381f11d-kube-api-access-mf475" (OuterVolumeSpecName: "kube-api-access-mf475") pod "40fa6e7c-2302-42c0-a379-541c4381f11d" (UID: "40fa6e7c-2302-42c0-a379-541c4381f11d"). InnerVolumeSpecName "kube-api-access-mf475". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:25:14.736098 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.736074    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74zpv\" (UniqueName: \"kubernetes.io/projected/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-kube-api-access-74zpv\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:25:14.736098 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.736095    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40fa6e7c-2302-42c0-a379-541c4381f11d-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:25:14.736261 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.736104    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11d995bb-b1a3-4c8a-8cd4-e093f8edc643-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:25:14.736261 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.736113    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf475\" (UniqueName: \"kubernetes.io/projected/40fa6e7c-2302-42c0-a379-541c4381f11d-kube-api-access-mf475\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:25:14.945582 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.945554    2576 generic.go:356] "Generic (PLEG): container finished" podID="40fa6e7c-2302-42c0-a379-541c4381f11d" containerID="dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07" exitCode=0
Jul 28 01:25:14.945744 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.945607    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"
Jul 28 01:25:14.945744 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.945642    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv" event={"ID":"40fa6e7c-2302-42c0-a379-541c4381f11d","Type":"ContainerDied","Data":"dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07"}
Jul 28 01:25:14.945744 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.945713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv" event={"ID":"40fa6e7c-2302-42c0-a379-541c4381f11d","Type":"ContainerDied","Data":"946d6cdfffd24490283c5743c9b57b96c5ef1b0909a756372a70fe0b88b78d55"}
Jul 28 01:25:14.945744 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.945743    2576 scope.go:117] "RemoveContainer" containerID="dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07"
Jul 28 01:25:14.947170 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.947101    2576 generic.go:356] "Generic (PLEG): container finished" podID="11d995bb-b1a3-4c8a-8cd4-e093f8edc643" containerID="19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe" exitCode=0
Jul 28 01:25:14.947170 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.947154    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"
Jul 28 01:25:14.947377 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.947153    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78" event={"ID":"11d995bb-b1a3-4c8a-8cd4-e093f8edc643","Type":"ContainerDied","Data":"19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe"}
Jul 28 01:25:14.947377 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.947293    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78" event={"ID":"11d995bb-b1a3-4c8a-8cd4-e093f8edc643","Type":"ContainerDied","Data":"40a706ad73f52e6d03d3676529e6b359374c85e57ee1a45c9800a8051037f115"}
Jul 28 01:25:14.954936 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.954918    2576 scope.go:117] "RemoveContainer" containerID="dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07"
Jul 28 01:25:14.955283 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:25:14.955257    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07\": container with ID starting with dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07 not found: ID does not exist" containerID="dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07"
Jul 28 01:25:14.955367 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.955293    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07"} err="failed to get container status \"dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07\": rpc error: code = NotFound desc = could not find container \"dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07\": container with ID starting with dbbdbbcd61e47c53f9f73ec004e03c3a907fdac6b00a039217f0ebf8c3d0aa07 not found: ID does not exist"
Jul 28 01:25:14.955367 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.955311    2576 scope.go:117] "RemoveContainer" containerID="19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe"
Jul 28 01:25:14.962649 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.962632    2576 scope.go:117] "RemoveContainer" containerID="19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe"
Jul 28 01:25:14.962875 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:25:14.962856    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe\": container with ID starting with 19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe not found: ID does not exist" containerID="19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe"
Jul 28 01:25:14.962925 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.962881    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe"} err="failed to get container status \"19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe\": rpc error: code = NotFound desc = could not find container \"19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe\": container with ID starting with 19ea34c75723803399c7c81c18bcfa6e0e1b1788cfb402712dfdf71ec1053dbe not found: ID does not exist"
Jul 28 01:25:14.971251 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.971214    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"]
Jul 28 01:25:14.975129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.975110    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-f1343b-56ccdfbcfb-n8n78"]
Jul 28 01:25:14.985387 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.985369    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"]
Jul 28 01:25:14.991540 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:14.991522    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-f1343b-75655cf4bb-7bjsv"]
Jul 28 01:25:15.751442 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:15.751409    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11d995bb-b1a3-4c8a-8cd4-e093f8edc643" path="/var/lib/kubelet/pods/11d995bb-b1a3-4c8a-8cd4-e093f8edc643/volumes"
Jul 28 01:25:15.751952 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:15.751931    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40fa6e7c-2302-42c0-a379-541c4381f11d" path="/var/lib/kubelet/pods/40fa6e7c-2302-42c0-a379-541c4381f11d/volumes"
Jul 28 01:25:18.757267 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:18.757212    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:19.123149 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:19.123120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:24.992305 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:24.992267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:25.054757 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:25.054724    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:25.244388 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:25.244303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:25.482893 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:25.482830    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:25.810517 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:25.810478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:26.114720 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:26.114639    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:26.727548 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:26.727490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:29.406438 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:29.406349    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:29.512339 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:29.512298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:29.613306 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:29.613275    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:29.740579 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:29.740501    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:30.128684 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.128646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:30.228982 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.228946    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:30.270006 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.269973    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"]
Jul 28 01:25:30.270310 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270297    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5c95fda-7a44-4744-9d80-23d171bb4cee" containerName="payload-pre-processing"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270311    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5c95fda-7a44-4744-9d80-23d171bb4cee" containerName="payload-pre-processing"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270323    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40fa6e7c-2302-42c0-a379-541c4381f11d" containerName="payload-processing"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270328    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="40fa6e7c-2302-42c0-a379-541c4381f11d" containerName="payload-processing"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270340    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7537664e-242e-45ef-b2c6-0c88eba7d697" containerName="payload-processing"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270345    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7537664e-242e-45ef-b2c6-0c88eba7d697" containerName="payload-processing"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270352    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6f750fa-4415-45a3-a4a4-ff4af601404d" containerName="main"
Jul 28 01:25:30.270359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270358    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6f750fa-4415-45a3-a4a4-ff4af601404d" containerName="main"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270366    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11d995bb-b1a3-4c8a-8cd4-e093f8edc643" containerName="payload-pre-processing"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270371    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="11d995bb-b1a3-4c8a-8cd4-e093f8edc643" containerName="payload-pre-processing"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270433    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="11d995bb-b1a3-4c8a-8cd4-e093f8edc643" containerName="payload-pre-processing"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270443    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6f750fa-4415-45a3-a4a4-ff4af601404d" containerName="main"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270450    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7537664e-242e-45ef-b2c6-0c88eba7d697" containerName="payload-processing"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270456    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="40fa6e7c-2302-42c0-a379-541c4381f11d" containerName="payload-processing"
Jul 28 01:25:30.270580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.270462    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5c95fda-7a44-4744-9d80-23d171bb4cee" containerName="payload-pre-processing"
Jul 28 01:25:30.274854 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.274836    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.277686 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.277665    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-5222f2-dockercfg-k5qpz\""
Jul 28 01:25:30.277762 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.277748    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5222f2\""
Jul 28 01:25:30.287413 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.287393    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"]
Jul 28 01:25:30.310249 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.310208    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"]
Jul 28 01:25:30.313480 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.313465    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.321083 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.321060    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"]
Jul 28 01:25:30.368813 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.368784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jft7t\" (UniqueName: \"kubernetes.io/projected/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-kube-api-access-jft7t\") pod \"payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") " pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.368923 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.368888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") " pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.469352 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.469284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a152fac1-ea62-4c10-9be5-ea7283c83d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.469674 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.469388    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") " pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.469674 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.469422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4q88\" (UniqueName: \"kubernetes.io/projected/a152fac1-ea62-4c10-9be5-ea7283c83d56-kube-api-access-l4q88\") pod \"payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.469674 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.469450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jft7t\" (UniqueName: \"kubernetes.io/projected/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-kube-api-access-jft7t\") pod \"payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") " pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.470086 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.470067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") " pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.477692 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.477669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jft7t\" (UniqueName: \"kubernetes.io/projected/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-kube-api-access-jft7t\") pod \"payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") " pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.570063 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.570025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a152fac1-ea62-4c10-9be5-ea7283c83d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.570261 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.570140    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4q88\" (UniqueName: \"kubernetes.io/projected/a152fac1-ea62-4c10-9be5-ea7283c83d56-kube-api-access-l4q88\") pod \"payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.570743 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.570720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a152fac1-ea62-4c10-9be5-ea7283c83d56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.582207 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.582175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4q88\" (UniqueName: \"kubernetes.io/projected/a152fac1-ea62-4c10-9be5-ea7283c83d56-kube-api-access-l4q88\") pod \"payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.585053 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.585034    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:30.624555 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.624342    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:30.750641 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.750583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"]
Jul 28 01:25:30.758546 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:25:30.758485    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c3ad9f_7dac_4411_ae87_4d95b5da4c7b.slice/crio-df9d25a0e4e160e989ac0d5b39b85bf676471afacb4c4acc59d3a1d5540f5959 WatchSource:0}: Error finding container df9d25a0e4e160e989ac0d5b39b85bf676471afacb4c4acc59d3a1d5540f5959: Status 404 returned error can't find the container with id df9d25a0e4e160e989ac0d5b39b85bf676471afacb4c4acc59d3a1d5540f5959
Jul 28 01:25:30.797891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:30.797869    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"]
Jul 28 01:25:30.809377 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:25:30.809355    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda152fac1_ea62_4c10_9be5_ea7283c83d56.slice/crio-224857eded524e815ad06610805809511a04eeb3483d4e56ca413c8fc829ca34 WatchSource:0}: Error finding container 224857eded524e815ad06610805809511a04eeb3483d4e56ca413c8fc829ca34: Status 404 returned error can't find the container with id 224857eded524e815ad06610805809511a04eeb3483d4e56ca413c8fc829ca34
Jul 28 01:25:31.032380 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.032298    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x" event={"ID":"a152fac1-ea62-4c10-9be5-ea7283c83d56","Type":"ContainerStarted","Data":"0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43"}
Jul 28 01:25:31.032380 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.032348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x" event={"ID":"a152fac1-ea62-4c10-9be5-ea7283c83d56","Type":"ContainerStarted","Data":"224857eded524e815ad06610805809511a04eeb3483d4e56ca413c8fc829ca34"}
Jul 28 01:25:31.032581 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.032405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:31.033929 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.033905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87" event={"ID":"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b","Type":"ContainerStarted","Data":"86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f"}
Jul 28 01:25:31.034055 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.033944    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87" event={"ID":"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b","Type":"ContainerStarted","Data":"df9d25a0e4e160e989ac0d5b39b85bf676471afacb4c4acc59d3a1d5540f5959"}
Jul 28 01:25:31.034055 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.033990    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:31.049619 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.049569    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x" podStartSLOduration=1.04955255 podStartE2EDuration="1.04955255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:31.047920501 +0000 UTC m=+2335.797059225" watchObservedRunningTime="2026-07-28 01:25:31.04955255 +0000 UTC m=+2335.798691273"
Jul 28 01:25:31.066612 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:31.066569    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87" podStartSLOduration=1.066557674 podStartE2EDuration="1.066557674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:31.063939383 +0000 UTC m=+2335.813078122" watchObservedRunningTime="2026-07-28 01:25:31.066557674 +0000 UTC m=+2335.815696398"
Jul 28 01:25:33.945384 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:33.945336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:34.604406 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:34.604378    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:34.733349 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:34.733316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:34.974905 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:34.974819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:37.915057 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:37.915018    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:38.003310 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.003281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:38.076103 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.076078    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:38.284444 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.284363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:38.573325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.573297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:38.672986 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.672953    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"]
Jul 28 01:25:38.677624 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.677607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.681018 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.680995    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-9c8614-dockercfg-q8kv2\""
Jul 28 01:25:38.681141 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.681057    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-9c8614\""
Jul 28 01:25:38.690639 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.690618    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"]
Jul 28 01:25:38.702290 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.702264    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"]
Jul 28 01:25:38.706194 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.706175    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.716286 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.716263    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"]
Jul 28 01:25:38.735699 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.735675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.735853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.735710    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhcqs\" (UniqueName: \"kubernetes.io/projected/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-kube-api-access-bhcqs\") pod \"payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.755686 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.755661    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:38.836730 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.836657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849324f-6897-422c-b75b-20290725451d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.836730 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.836691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.836730 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.836714    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhcqs\" (UniqueName: \"kubernetes.io/projected/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-kube-api-access-bhcqs\") pod \"payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.836947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.836836    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krmk5\" (UniqueName: \"kubernetes.io/projected/3849324f-6897-422c-b75b-20290725451d-kube-api-access-krmk5\") pod \"payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.837421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.837404    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-plugins-config-volume\") pod \"payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.846168 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.846143    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhcqs\" (UniqueName: \"kubernetes.io/projected/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-kube-api-access-bhcqs\") pod \"payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") " pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:38.937265 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.937236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krmk5\" (UniqueName: \"kubernetes.io/projected/3849324f-6897-422c-b75b-20290725451d-kube-api-access-krmk5\") pod \"payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.937583 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.937304    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849324f-6897-422c-b75b-20290725451d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.937926 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.937907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849324f-6897-422c-b75b-20290725451d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.946151 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.946121    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krmk5\" (UniqueName: \"kubernetes.io/projected/3849324f-6897-422c-b75b-20290725451d-kube-api-access-krmk5\") pod \"payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:38.989179 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:38.989152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:39.017429 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:39.017405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:39.343560 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:39.343520    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"]
Jul 28 01:25:39.346979 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:25:39.346941    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3849324f_6897_422c_b75b_20290725451d.slice/crio-e1c7e68470d9a065a8a221aea59e323d78320dc261cca78bc683f509ddc94b6c WatchSource:0}: Error finding container e1c7e68470d9a065a8a221aea59e323d78320dc261cca78bc683f509ddc94b6c: Status 404 returned error can't find the container with id e1c7e68470d9a065a8a221aea59e323d78320dc261cca78bc683f509ddc94b6c
Jul 28 01:25:39.372365 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:39.372342    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"]
Jul 28 01:25:39.379213 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:25:39.379190    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdea3c66_2db5_48ef_88c5_7f888e5ce9dd.slice/crio-bee9460a8e8237c37686afd62e3a18464d52b74d31cf4fa7b72d7e4d6dfc96f9 WatchSource:0}: Error finding container bee9460a8e8237c37686afd62e3a18464d52b74d31cf4fa7b72d7e4d6dfc96f9: Status 404 returned error can't find the container with id bee9460a8e8237c37686afd62e3a18464d52b74d31cf4fa7b72d7e4d6dfc96f9
Jul 28 01:25:40.080491 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.080435    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh" event={"ID":"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd","Type":"ContainerStarted","Data":"823f7c7a098a286882af80c3f7f51ea9da0f7af549d2418cf655eedea4b45097"}
Jul 28 01:25:40.080491 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.080494    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh" event={"ID":"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd","Type":"ContainerStarted","Data":"bee9460a8e8237c37686afd62e3a18464d52b74d31cf4fa7b72d7e4d6dfc96f9"}
Jul 28 01:25:40.081015 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.080622    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:40.081875 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.081850    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f" event={"ID":"3849324f-6897-422c-b75b-20290725451d","Type":"ContainerStarted","Data":"43404f2f7fe0502d6943b4469cb1d832e121efb1db0ee0b5e17b0ee9f434b377"}
Jul 28 01:25:40.081959 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.081898    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f" event={"ID":"3849324f-6897-422c-b75b-20290725451d","Type":"ContainerStarted","Data":"e1c7e68470d9a065a8a221aea59e323d78320dc261cca78bc683f509ddc94b6c"}
Jul 28 01:25:40.081959 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.081923    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:40.100092 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.100052    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh" podStartSLOduration=2.100039143 podStartE2EDuration="2.100039143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:40.099114086 +0000 UTC m=+2344.848252811" watchObservedRunningTime="2026-07-28 01:25:40.100039143 +0000 UTC m=+2344.849177864"
Jul 28 01:25:40.114526 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:40.114484    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f" podStartSLOduration=2.114469794 podStartE2EDuration="2.114469794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:40.113785078 +0000 UTC m=+2344.862923804" watchObservedRunningTime="2026-07-28 01:25:40.114469794 +0000 UTC m=+2344.863608518"
Jul 28 01:25:42.041846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:42.041818    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:25:42.042478 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:42.041896    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:25:42.431407 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:42.431377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:25:50.449734 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.449698    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"]
Jul 28 01:25:50.453158 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.453138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:50.455937 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.455914    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:25:50.456058 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.455912    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:25:50.462054 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.462027    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"]
Jul 28 01:25:50.533340 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.533313    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790449c-6599-4d99-8ba4-0b1449687cd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5222f2-wk874\" (UID: \"d790449c-6599-4d99-8ba4-0b1449687cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:50.634015 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.633981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790449c-6599-4d99-8ba4-0b1449687cd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5222f2-wk874\" (UID: \"d790449c-6599-4d99-8ba4-0b1449687cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:50.634596 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.634576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790449c-6599-4d99-8ba4-0b1449687cd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5222f2-wk874\" (UID: \"d790449c-6599-4d99-8ba4-0b1449687cd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:50.764772 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.764712    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:50.904565 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.904540    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"]
Jul 28 01:25:50.906008 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:25:50.905978    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd790449c_6599_4d99_8ba4_0b1449687cd9.slice/crio-126cb49bb278f7b3184a6b3c5d1e5103461c66f3ac2b0eb46efa79681c61c0ed WatchSource:0}: Error finding container 126cb49bb278f7b3184a6b3c5d1e5103461c66f3ac2b0eb46efa79681c61c0ed: Status 404 returned error can't find the container with id 126cb49bb278f7b3184a6b3c5d1e5103461c66f3ac2b0eb46efa79681c61c0ed
Jul 28 01:25:50.907926 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:50.907908    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:51.088558 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:51.088535    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:25:51.088702 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:51.088595    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:25:51.129665 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:51.129637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874" event={"ID":"d790449c-6599-4d99-8ba4-0b1449687cd9","Type":"ContainerStarted","Data":"126cb49bb278f7b3184a6b3c5d1e5103461c66f3ac2b0eb46efa79681c61c0ed"}
Jul 28 01:25:55.154436 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:55.154391    2576 generic.go:356] "Generic (PLEG): container finished" podID="d790449c-6599-4d99-8ba4-0b1449687cd9" containerID="9b0f7fc4550d42dba73040693f8ff27a3e36def22efd8dbb0764695d8a24ebe4" exitCode=0
Jul 28 01:25:55.154888 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:55.154479    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874" event={"ID":"d790449c-6599-4d99-8ba4-0b1449687cd9","Type":"ContainerDied","Data":"9b0f7fc4550d42dba73040693f8ff27a3e36def22efd8dbb0764695d8a24ebe4"}
Jul 28 01:25:56.294639 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:56.294618    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:56.387967 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:56.387938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790449c-6599-4d99-8ba4-0b1449687cd9-maas-api-service-ca\") pod \"d790449c-6599-4d99-8ba4-0b1449687cd9\" (UID: \"d790449c-6599-4d99-8ba4-0b1449687cd9\") "
Jul 28 01:25:56.388294 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:56.388269    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d790449c-6599-4d99-8ba4-0b1449687cd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d790449c-6599-4d99-8ba4-0b1449687cd9" (UID: "d790449c-6599-4d99-8ba4-0b1449687cd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:25:56.489001 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:56.488930    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d790449c-6599-4d99-8ba4-0b1449687cd9-maas-api-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:25:57.164211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:57.164171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874" event={"ID":"d790449c-6599-4d99-8ba4-0b1449687cd9","Type":"ContainerDied","Data":"126cb49bb278f7b3184a6b3c5d1e5103461c66f3ac2b0eb46efa79681c61c0ed"}
Jul 28 01:25:57.164211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:57.164211    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"
Jul 28 01:25:57.164432 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:25:57.164245    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="126cb49bb278f7b3184a6b3c5d1e5103461c66f3ac2b0eb46efa79681c61c0ed"
Jul 28 01:26:06.318504 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.318466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:06.340564 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.340532    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"]
Jul 28 01:26:06.340819 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.340775    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87" podUID="d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" containerName="payload-processing" containerID="cri-o://86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f" gracePeriod=30
Jul 28 01:26:06.349176 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.349150    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"]
Jul 28 01:26:06.349628 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.349601    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x" podUID="a152fac1-ea62-4c10-9be5-ea7283c83d56" containerName="payload-pre-processing" containerID="cri-o://0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43" gracePeriod=30
Jul 28 01:26:06.422368 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.422339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:06.518986 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.518954    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:06.615855 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.615817    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:26:06.618805 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.618786    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:26:06.677675 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.677642    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a152fac1-ea62-4c10-9be5-ea7283c83d56-plugins-config-volume\") pod \"a152fac1-ea62-4c10-9be5-ea7283c83d56\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") "
Jul 28 01:26:06.677675 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.677681    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4q88\" (UniqueName: \"kubernetes.io/projected/a152fac1-ea62-4c10-9be5-ea7283c83d56-kube-api-access-l4q88\") pod \"a152fac1-ea62-4c10-9be5-ea7283c83d56\" (UID: \"a152fac1-ea62-4c10-9be5-ea7283c83d56\") "
Jul 28 01:26:06.677878 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.677716    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jft7t\" (UniqueName: \"kubernetes.io/projected/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-kube-api-access-jft7t\") pod \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") "
Jul 28 01:26:06.677878 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.677756    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-plugins-config-volume\") pod \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\" (UID: \"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b\") "
Jul 28 01:26:06.678019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.677996    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a152fac1-ea62-4c10-9be5-ea7283c83d56-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a152fac1-ea62-4c10-9be5-ea7283c83d56" (UID: "a152fac1-ea62-4c10-9be5-ea7283c83d56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:06.678083 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.678060    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a152fac1-ea62-4c10-9be5-ea7283c83d56-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:06.678256 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.678213    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" (UID: "d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:06.679777 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.679756    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-kube-api-access-jft7t" (OuterVolumeSpecName: "kube-api-access-jft7t") pod "d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" (UID: "d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b"). InnerVolumeSpecName "kube-api-access-jft7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:06.679822 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.679763    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a152fac1-ea62-4c10-9be5-ea7283c83d56-kube-api-access-l4q88" (OuterVolumeSpecName: "kube-api-access-l4q88") pod "a152fac1-ea62-4c10-9be5-ea7283c83d56" (UID: "a152fac1-ea62-4c10-9be5-ea7283c83d56"). InnerVolumeSpecName "kube-api-access-l4q88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:06.778636 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.778600    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4q88\" (UniqueName: \"kubernetes.io/projected/a152fac1-ea62-4c10-9be5-ea7283c83d56-kube-api-access-l4q88\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:06.778636 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.778630    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jft7t\" (UniqueName: \"kubernetes.io/projected/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-kube-api-access-jft7t\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:06.778636 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:06.778639    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:07.207504 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.207468    2576 generic.go:356] "Generic (PLEG): container finished" podID="a152fac1-ea62-4c10-9be5-ea7283c83d56" containerID="0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43" exitCode=0
Jul 28 01:26:07.207701 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.207519    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"
Jul 28 01:26:07.207701 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.207562    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x" event={"ID":"a152fac1-ea62-4c10-9be5-ea7283c83d56","Type":"ContainerDied","Data":"0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43"}
Jul 28 01:26:07.207701 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.207628    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x" event={"ID":"a152fac1-ea62-4c10-9be5-ea7283c83d56","Type":"ContainerDied","Data":"224857eded524e815ad06610805809511a04eeb3483d4e56ca413c8fc829ca34"}
Jul 28 01:26:07.207701 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.207653    2576 scope.go:117] "RemoveContainer" containerID="0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43"
Jul 28 01:26:07.208946 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.208922    2576 generic.go:356] "Generic (PLEG): container finished" podID="d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" containerID="86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f" exitCode=0
Jul 28 01:26:07.209078 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.208964    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"
Jul 28 01:26:07.209078 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.209000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87" event={"ID":"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b","Type":"ContainerDied","Data":"86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f"}
Jul 28 01:26:07.209078 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.209047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87" event={"ID":"d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b","Type":"ContainerDied","Data":"df9d25a0e4e160e989ac0d5b39b85bf676471afacb4c4acc59d3a1d5540f5959"}
Jul 28 01:26:07.216683 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.216667    2576 scope.go:117] "RemoveContainer" containerID="0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43"
Jul 28 01:26:07.216968 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:26:07.216946    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43\": container with ID starting with 0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43 not found: ID does not exist" containerID="0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43"
Jul 28 01:26:07.217064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.216974    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43"} err="failed to get container status \"0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43\": rpc error: code = NotFound desc = could not find container \"0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43\": container with ID starting with 0ca2f154e27d71581888d64d7c9dc19f124c56610f5880bdee7e2bda5377ab43 not found: ID does not exist"
Jul 28 01:26:07.217064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.216991    2576 scope.go:117] "RemoveContainer" containerID="86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f"
Jul 28 01:26:07.225116 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.225101    2576 scope.go:117] "RemoveContainer" containerID="86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f"
Jul 28 01:26:07.225387 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:26:07.225361    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f\": container with ID starting with 86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f not found: ID does not exist" containerID="86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f"
Jul 28 01:26:07.225440 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.225387    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f"} err="failed to get container status \"86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f\": rpc error: code = NotFound desc = could not find container \"86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f\": container with ID starting with 86e5f7d4550dcb6bb3959ef31281e6ce12f769c1691422ad94e616a5de10ac8f not found: ID does not exist"
Jul 28 01:26:07.234000 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.233979    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"]
Jul 28 01:26:07.237002 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.236984    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-5222f2-6fdb57db94-hnl87"]
Jul 28 01:26:07.246970 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.246950    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"]
Jul 28 01:26:07.251300 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.251281    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5222f2-56675d7ccb-xfz7x"]
Jul 28 01:26:07.750796 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.750755    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a152fac1-ea62-4c10-9be5-ea7283c83d56" path="/var/lib/kubelet/pods/a152fac1-ea62-4c10-9be5-ea7283c83d56/volumes"
Jul 28 01:26:07.751315 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:07.751294    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" path="/var/lib/kubelet/pods/d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b/volumes"
Jul 28 01:26:10.545164 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:10.545115    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:10.661899 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:10.661867    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"]
Jul 28 01:26:10.664859 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:10.664834    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5222f2-wk874"]
Jul 28 01:26:10.973657 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:10.973629    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:11.757645 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:11.757607    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d790449c-6599-4d99-8ba4-0b1449687cd9" path="/var/lib/kubelet/pods/d790449c-6599-4d99-8ba4-0b1449687cd9/volumes"
Jul 28 01:26:15.280867 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:15.280835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:16.894829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:16.894791    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:16.969545 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:16.969515    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.302811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:17.302722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.451931 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:17.451892    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.092981 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.092951    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.114741 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.114714    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"]
Jul 28 01:26:33.114975 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.114952    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh" podUID="fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" containerName="payload-processing" containerID="cri-o://823f7c7a098a286882af80c3f7f51ea9da0f7af549d2418cf655eedea4b45097" gracePeriod=30
Jul 28 01:26:33.120622 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.120595    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"]
Jul 28 01:26:33.120861 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.120808    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f" podUID="3849324f-6897-422c-b75b-20290725451d" containerName="payload-pre-processing" containerID="cri-o://43404f2f7fe0502d6943b4469cb1d832e121efb1db0ee0b5e17b0ee9f434b377" gracePeriod=30
Jul 28 01:26:33.194236 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.194196    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.274913 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.274880    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.332843 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.332817    2576 generic.go:356] "Generic (PLEG): container finished" podID="fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" containerID="823f7c7a098a286882af80c3f7f51ea9da0f7af549d2418cf655eedea4b45097" exitCode=0
Jul 28 01:26:33.333001 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.332903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh" event={"ID":"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd","Type":"ContainerDied","Data":"823f7c7a098a286882af80c3f7f51ea9da0f7af549d2418cf655eedea4b45097"}
Jul 28 01:26:33.334367 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.334343    2576 generic.go:356] "Generic (PLEG): container finished" podID="3849324f-6897-422c-b75b-20290725451d" containerID="43404f2f7fe0502d6943b4469cb1d832e121efb1db0ee0b5e17b0ee9f434b377" exitCode=0
Jul 28 01:26:33.334495 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.334426    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f" event={"ID":"3849324f-6897-422c-b75b-20290725451d","Type":"ContainerDied","Data":"43404f2f7fe0502d6943b4469cb1d832e121efb1db0ee0b5e17b0ee9f434b377"}
Jul 28 01:26:33.364850 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.364829    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:26:33.390268 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.390245    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:26:33.401154 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.401134    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-plugins-config-volume\") pod \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") "
Jul 28 01:26:33.401258 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.401163    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849324f-6897-422c-b75b-20290725451d-plugins-config-volume\") pod \"3849324f-6897-422c-b75b-20290725451d\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") "
Jul 28 01:26:33.401258 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.401179    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krmk5\" (UniqueName: \"kubernetes.io/projected/3849324f-6897-422c-b75b-20290725451d-kube-api-access-krmk5\") pod \"3849324f-6897-422c-b75b-20290725451d\" (UID: \"3849324f-6897-422c-b75b-20290725451d\") "
Jul 28 01:26:33.401258 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.401246    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhcqs\" (UniqueName: \"kubernetes.io/projected/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-kube-api-access-bhcqs\") pod \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\" (UID: \"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd\") "
Jul 28 01:26:33.401540 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.401518    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" (UID: "fdea3c66-2db5-48ef-88c5-7f888e5ce9dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:33.401595 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.401550    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3849324f-6897-422c-b75b-20290725451d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3849324f-6897-422c-b75b-20290725451d" (UID: "3849324f-6897-422c-b75b-20290725451d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:26:33.403266 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.403234    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-kube-api-access-bhcqs" (OuterVolumeSpecName: "kube-api-access-bhcqs") pod "fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" (UID: "fdea3c66-2db5-48ef-88c5-7f888e5ce9dd"). InnerVolumeSpecName "kube-api-access-bhcqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:33.403266 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.403252    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3849324f-6897-422c-b75b-20290725451d-kube-api-access-krmk5" (OuterVolumeSpecName: "kube-api-access-krmk5") pod "3849324f-6897-422c-b75b-20290725451d" (UID: "3849324f-6897-422c-b75b-20290725451d"). InnerVolumeSpecName "kube-api-access-krmk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:26:33.502369 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.502333    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhcqs\" (UniqueName: \"kubernetes.io/projected/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-kube-api-access-bhcqs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:33.502369 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.502362    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:33.502369 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.502371    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3849324f-6897-422c-b75b-20290725451d-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:33.502369 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:33.502380    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krmk5\" (UniqueName: \"kubernetes.io/projected/3849324f-6897-422c-b75b-20290725451d-kube-api-access-krmk5\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:26:34.340651 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.340621    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"
Jul 28 01:26:34.341072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.340620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh" event={"ID":"fdea3c66-2db5-48ef-88c5-7f888e5ce9dd","Type":"ContainerDied","Data":"bee9460a8e8237c37686afd62e3a18464d52b74d31cf4fa7b72d7e4d6dfc96f9"}
Jul 28 01:26:34.341072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.340768    2576 scope.go:117] "RemoveContainer" containerID="823f7c7a098a286882af80c3f7f51ea9da0f7af549d2418cf655eedea4b45097"
Jul 28 01:26:34.342095 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.342075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f" event={"ID":"3849324f-6897-422c-b75b-20290725451d","Type":"ContainerDied","Data":"e1c7e68470d9a065a8a221aea59e323d78320dc261cca78bc683f509ddc94b6c"}
Jul 28 01:26:34.342173 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.342129    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"
Jul 28 01:26:34.349210 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.349191    2576 scope.go:117] "RemoveContainer" containerID="43404f2f7fe0502d6943b4469cb1d832e121efb1db0ee0b5e17b0ee9f434b377"
Jul 28 01:26:34.359442 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.359419    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"]
Jul 28 01:26:34.363971 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.363950    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-9c8614-7678bbbf48-fmhzh"]
Jul 28 01:26:34.375592 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.375572    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"]
Jul 28 01:26:34.381992 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:34.381973    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-9c8614-7679747784-bvb9f"]
Jul 28 01:26:35.751396 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:35.751359    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3849324f-6897-422c-b75b-20290725451d" path="/var/lib/kubelet/pods/3849324f-6897-422c-b75b-20290725451d/volumes"
Jul 28 01:26:35.751878 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:35.751857    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" path="/var/lib/kubelet/pods/fdea3c66-2db5-48ef-88c5-7f888e5ce9dd/volumes"
Jul 28 01:26:37.665756 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:37.665723    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:38.097295 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:38.097258    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.043485 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:44.043455    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.144837 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:44.144799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.308697 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:44.308665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.535252 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:44.535207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.859984 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:44.859944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.166450 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:45.166369    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.602658 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:45.602626    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.737399 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:47.737363    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.833650 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:47.833618    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.913111 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:47.913082    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.188383 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.188347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.414718 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.414688    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.512689 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.512608    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:48.518014 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.517990    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"]
Jul 28 01:26:48.518358 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518344    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" containerName="payload-processing"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518360    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" containerName="payload-processing"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518369    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d790449c-6599-4d99-8ba4-0b1449687cd9" containerName="revoke-keys"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518375    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d790449c-6599-4d99-8ba4-0b1449687cd9" containerName="revoke-keys"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518384    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a152fac1-ea62-4c10-9be5-ea7283c83d56" containerName="payload-pre-processing"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518391    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a152fac1-ea62-4c10-9be5-ea7283c83d56" containerName="payload-pre-processing"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" containerName="payload-processing"
Jul 28 01:26:48.518421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518411    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" containerName="payload-processing"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518431    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3849324f-6897-422c-b75b-20290725451d" containerName="payload-pre-processing"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518436    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3849324f-6897-422c-b75b-20290725451d" containerName="payload-pre-processing"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518494    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d790449c-6599-4d99-8ba4-0b1449687cd9" containerName="revoke-keys"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518502    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a152fac1-ea62-4c10-9be5-ea7283c83d56" containerName="payload-pre-processing"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518509    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3c3ad9f-7dac-4411-ae87-4d95b5da4c7b" containerName="payload-processing"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518514    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdea3c66-2db5-48ef-88c5-7f888e5ce9dd" containerName="payload-processing"
Jul 28 01:26:48.518673 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.518521    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3849324f-6897-422c-b75b-20290725451d" containerName="payload-pre-processing"
Jul 28 01:26:48.521051 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.521033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.524330 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.524309    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-14a1d0\""
Jul 28 01:26:48.524450 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.524342    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-14a1d0-dockercfg-7s44d\""
Jul 28 01:26:48.536039 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.536014    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"]
Jul 28 01:26:48.549895 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.549866    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"]
Jul 28 01:26:48.554322 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.554303    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.565182 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.565163    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"]
Jul 28 01:26:48.623914 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.623888    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.624030 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.623942    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6fkm\" (UniqueName: \"kubernetes.io/projected/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-kube-api-access-r6fkm\") pod \"payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.624078 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.624028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjc8q\" (UniqueName: \"kubernetes.io/projected/ed127e22-f2df-44a2-b740-44184320c13d-kube-api-access-gjc8q\") pod \"payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.624133 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.624084    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127e22-f2df-44a2-b740-44184320c13d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.725099 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.725067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjc8q\" (UniqueName: \"kubernetes.io/projected/ed127e22-f2df-44a2-b740-44184320c13d-kube-api-access-gjc8q\") pod \"payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.725273 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.725132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127e22-f2df-44a2-b740-44184320c13d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.725273 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.725177    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.725409 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.725305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6fkm\" (UniqueName: \"kubernetes.io/projected/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-kube-api-access-r6fkm\") pod \"payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.725893 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.725872    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127e22-f2df-44a2-b740-44184320c13d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.725971 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.725873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.746639 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.746612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6fkm\" (UniqueName: \"kubernetes.io/projected/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-kube-api-access-r6fkm\") pod \"payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.747804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.747777    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjc8q\" (UniqueName: \"kubernetes.io/projected/ed127e22-f2df-44a2-b740-44184320c13d-kube-api-access-gjc8q\") pod \"payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.831571 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.831547    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:48.864563 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.864540    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:48.996123 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:48.996096    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"]
Jul 28 01:26:48.998009 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:26:48.997981    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded127e22_f2df_44a2_b740_44184320c13d.slice/crio-093e4a0405ffbba1e59a381eff1a2dd31599f04a39a18f250e407e8414c016cb WatchSource:0}: Error finding container 093e4a0405ffbba1e59a381eff1a2dd31599f04a39a18f250e407e8414c016cb: Status 404 returned error can't find the container with id 093e4a0405ffbba1e59a381eff1a2dd31599f04a39a18f250e407e8414c016cb
Jul 28 01:26:49.014430 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.014399    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"]
Jul 28 01:26:49.016767 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:26:49.016733    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc85c5ea7_6b2d_4d0b_a1f4_a17143a62bb4.slice/crio-d04517db86de5ee46417d1e8f81b50fed0cac2af4aa577a2eff653097d368f24 WatchSource:0}: Error finding container d04517db86de5ee46417d1e8f81b50fed0cac2af4aa577a2eff653097d368f24: Status 404 returned error can't find the container with id d04517db86de5ee46417d1e8f81b50fed0cac2af4aa577a2eff653097d368f24
Jul 28 01:26:49.411019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.410937    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2" event={"ID":"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4","Type":"ContainerStarted","Data":"c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248"}
Jul 28 01:26:49.411019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.410990    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2" event={"ID":"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4","Type":"ContainerStarted","Data":"d04517db86de5ee46417d1e8f81b50fed0cac2af4aa577a2eff653097d368f24"}
Jul 28 01:26:49.411019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.411020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:26:49.412484 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.412461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx" event={"ID":"ed127e22-f2df-44a2-b740-44184320c13d","Type":"ContainerStarted","Data":"e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262"}
Jul 28 01:26:49.412586 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.412502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx" event={"ID":"ed127e22-f2df-44a2-b740-44184320c13d","Type":"ContainerStarted","Data":"093e4a0405ffbba1e59a381eff1a2dd31599f04a39a18f250e407e8414c016cb"}
Jul 28 01:26:49.412586 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.412548    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:26:49.430772 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.430732    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2" podStartSLOduration=1.430720846 podStartE2EDuration="1.430720846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:49.430173262 +0000 UTC m=+2414.179311988" watchObservedRunningTime="2026-07-28 01:26:49.430720846 +0000 UTC m=+2414.179859569"
Jul 28 01:26:49.449313 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:49.449272    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx" podStartSLOduration=1.449259846 podStartE2EDuration="1.449259846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:49.447752516 +0000 UTC m=+2414.196891240" watchObservedRunningTime="2026-07-28 01:26:49.449259846 +0000 UTC m=+2414.198398571"
Jul 28 01:26:52.210838 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:52.210788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.559357 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:52.559320    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.874853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:52.874763    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.979759 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:52.979706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.971743 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:53.971711    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.354173 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:55.354132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.457921 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:55.457890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.633591 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:55.633503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.765087 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:55.765040    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.091272 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.091243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.195055 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.195002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.196097 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.196073    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"]
Jul 28 01:26:56.199553 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.199535    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.202212 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.202189    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-3131e3-dockercfg-zjl4f\""
Jul 28 01:26:56.202331 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.202237    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-3131e3\""
Jul 28 01:26:56.209298 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.209274    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"]
Jul 28 01:26:56.215479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.215456    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"]
Jul 28 01:26:56.218660 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.218644    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.240999 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.240975    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"]
Jul 28 01:26:56.294665 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.294641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820afed0-72bd-42e0-922a-2af06a4fc493-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") " pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.294808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.294672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfbw\" (UniqueName: \"kubernetes.io/projected/820afed0-72bd-42e0-922a-2af06a4fc493-kube-api-access-jmfbw\") pod \"payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") " pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.294808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.294715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.294808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.294739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7mfq\" (UniqueName: \"kubernetes.io/projected/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-kube-api-access-j7mfq\") pod \"payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.395768 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.395695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820afed0-72bd-42e0-922a-2af06a4fc493-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") " pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.395768 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.395727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfbw\" (UniqueName: \"kubernetes.io/projected/820afed0-72bd-42e0-922a-2af06a4fc493-kube-api-access-jmfbw\") pod \"payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") " pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.396154 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.395774    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.396154 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.395797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7mfq\" (UniqueName: \"kubernetes.io/projected/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-kube-api-access-j7mfq\") pod \"payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.396462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.396443    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820afed0-72bd-42e0-922a-2af06a4fc493-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") " pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.396566 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.396548    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.404386 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.404362    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7mfq\" (UniqueName: \"kubernetes.io/projected/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-kube-api-access-j7mfq\") pod \"payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.404544 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.404525    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfbw\" (UniqueName: \"kubernetes.io/projected/820afed0-72bd-42e0-922a-2af06a4fc493-kube-api-access-jmfbw\") pod \"payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") " pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.511021 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.510997    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:56.528117 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.528096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:56.657996 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.657956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"]
Jul 28 01:26:56.659404 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:26:56.659377    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65f8d5ff_21fe_421c_8f3b_d16b4d90e891.slice/crio-c1413c40b373cff20d79fd885124ff80093a851085c48030b60a5349075d512d WatchSource:0}: Error finding container c1413c40b373cff20d79fd885124ff80093a851085c48030b60a5349075d512d: Status 404 returned error can't find the container with id c1413c40b373cff20d79fd885124ff80093a851085c48030b60a5349075d512d
Jul 28 01:26:56.889007 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:56.888983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"]
Jul 28 01:26:56.890857 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:26:56.890831    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820afed0_72bd_42e0_922a_2af06a4fc493.slice/crio-fd3467dbb6f8eb72b9118035b51b7a47aba0da1a1dc89d02fcb0aca630021d0c WatchSource:0}: Error finding container fd3467dbb6f8eb72b9118035b51b7a47aba0da1a1dc89d02fcb0aca630021d0c: Status 404 returned error can't find the container with id fd3467dbb6f8eb72b9118035b51b7a47aba0da1a1dc89d02fcb0aca630021d0c
Jul 28 01:26:57.454477 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.454439    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v" event={"ID":"820afed0-72bd-42e0-922a-2af06a4fc493","Type":"ContainerStarted","Data":"457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8"}
Jul 28 01:26:57.454934 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.454540    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v" event={"ID":"820afed0-72bd-42e0-922a-2af06a4fc493","Type":"ContainerStarted","Data":"fd3467dbb6f8eb72b9118035b51b7a47aba0da1a1dc89d02fcb0aca630021d0c"}
Jul 28 01:26:57.454934 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.454569    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:26:57.456448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.456419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc" event={"ID":"65f8d5ff-21fe-421c-8f3b-d16b4d90e891","Type":"ContainerStarted","Data":"18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb"}
Jul 28 01:26:57.456578 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.456472    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc" event={"ID":"65f8d5ff-21fe-421c-8f3b-d16b4d90e891","Type":"ContainerStarted","Data":"c1413c40b373cff20d79fd885124ff80093a851085c48030b60a5349075d512d"}
Jul 28 01:26:57.456578 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.456500    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:26:57.472707 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.472669    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v" podStartSLOduration=1.47265521 podStartE2EDuration="1.47265521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:57.47051785 +0000 UTC m=+2422.219656576" watchObservedRunningTime="2026-07-28 01:26:57.47265521 +0000 UTC m=+2422.221793933"
Jul 28 01:26:57.486708 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:57.486667    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc" podStartSLOduration=1.486653666 podStartE2EDuration="1.486653666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:57.485388102 +0000 UTC m=+2422.234526825" watchObservedRunningTime="2026-07-28 01:26:57.486653666 +0000 UTC m=+2422.235792389"
Jul 28 01:26:59.891720 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:26:59.891645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:00.418968 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:00.418938    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:27:00.419163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:00.419005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:27:08.463506 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:08.463470    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:27:08.464004 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:08.463563    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:27:30.129428 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.129394    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"]
Jul 28 01:27:30.133342 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.133322    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.136215 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.136187    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"openshift-service-ca.crt\""
Jul 28 01:27:30.137516 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.137496    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"default-dockercfg-kvlf8\""
Jul 28 01:27:30.137741 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.137678    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"auth-test-model-14a1d0-kserve-self-signed-certs\""
Jul 28 01:27:30.137939 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.137697    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"kube-root-ca.crt\""
Jul 28 01:27:30.144642 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.144622    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"]
Jul 28 01:27:30.168470 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.168437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:30.179669 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.179646    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a345026-6ea0-4939-bc1c-cbc2fe599177-tls-certs\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.179794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.179684    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-home\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.179794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.179707    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-model-cache\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.179794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.179725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h2tj\" (UniqueName: \"kubernetes.io/projected/4a345026-6ea0-4939-bc1c-cbc2fe599177-kube-api-access-8h2tj\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.179917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.179799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-tmp-dir\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.179917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.179862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-dshm\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.280823 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.280790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-dshm\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.280991 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.280885    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a345026-6ea0-4939-bc1c-cbc2fe599177-tls-certs\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.280991 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.280925    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-home\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.280991 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.280953    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-model-cache\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.280991 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.280975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h2tj\" (UniqueName: \"kubernetes.io/projected/4a345026-6ea0-4939-bc1c-cbc2fe599177-kube-api-access-8h2tj\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.281234 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.281007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-tmp-dir\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.281609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.281558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-model-cache\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.281727 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.281629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-tmp-dir\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.282007 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.281978    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-home\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.283395 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.283369    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-dshm\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.283480 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.283399    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a345026-6ea0-4939-bc1c-cbc2fe599177-tls-certs\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.297199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.297175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h2tj\" (UniqueName: \"kubernetes.io/projected/4a345026-6ea0-4939-bc1c-cbc2fe599177-kube-api-access-8h2tj\") pod \"auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") " pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.444428 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.444345    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:30.579556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.579521    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"]
Jul 28 01:27:30.581933 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:27:30.581894    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a345026_6ea0_4939_bc1c_cbc2fe599177.slice/crio-be51f95f59f873efd09a3fbced0ebcb2e24416bc048e36e6a894b47b87716ac9 WatchSource:0}: Error finding container be51f95f59f873efd09a3fbced0ebcb2e24416bc048e36e6a894b47b87716ac9: Status 404 returned error can't find the container with id be51f95f59f873efd09a3fbced0ebcb2e24416bc048e36e6a894b47b87716ac9
Jul 28 01:27:30.608085 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:30.608062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b" event={"ID":"4a345026-6ea0-4939-bc1c-cbc2fe599177","Type":"ContainerStarted","Data":"be51f95f59f873efd09a3fbced0ebcb2e24416bc048e36e6a894b47b87716ac9"}
Jul 28 01:27:31.614762 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:31.614725    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b" event={"ID":"4a345026-6ea0-4939-bc1c-cbc2fe599177","Type":"ContainerStarted","Data":"32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc"}
Jul 28 01:27:31.631944 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:31.631837    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b" podStartSLOduration=1.631818754 podStartE2EDuration="1.631818754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:27:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:31.631651586 +0000 UTC m=+2456.380790311" watchObservedRunningTime="2026-07-28 01:27:31.631818754 +0000 UTC m=+2456.380957479"
Jul 28 01:27:40.444997 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:40.444961    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:40.445440 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:40.445018    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:40.457425 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:40.457402    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:27:40.670610 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:27:40.670580    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:28:16.851276 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:28:16.851243    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:22.514764 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:28:22.514723    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:22.574654 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:28:22.574616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:55.693914 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:29:55.693878    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:00.142740 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.142647    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"]
Jul 28 01:30:00.146019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.145997    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"]
Jul 28 01:30:00.146171 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.146147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"
Jul 28 01:30:00.148904 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.148880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:30:00.150083 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.150065    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"]
Jul 28 01:30:00.150176 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.150161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"
Jul 28 01:30:00.153618 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.153512    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"
Jul 28 01:30:00.157550 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.157528    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"
Jul 28 01:30:00.158994 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.158971    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"]
Jul 28 01:30:00.161994 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.161970    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"]
Jul 28 01:30:00.169186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.169168    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"
Jul 28 01:30:00.177310 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.177246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"]
Jul 28 01:30:00.205910 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.205876    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"
Jul 28 01:30:00.325247 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.325189    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"]
Jul 28 01:30:00.327657 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:30:00.327623    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb5d3941_844d_4964_866b_366c1ad6d34c.slice/crio-a1b0d302ad124d7b98b7a52fd38c7e6d53ff72c3a66437ac0a14fb8c97855ef4 WatchSource:0}: Error finding container a1b0d302ad124d7b98b7a52fd38c7e6d53ff72c3a66437ac0a14fb8c97855ef4: Status 404 returned error can't find the container with id a1b0d302ad124d7b98b7a52fd38c7e6d53ff72c3a66437ac0a14fb8c97855ef4
Jul 28 01:30:00.382510 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.382486    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"]
Jul 28 01:30:00.384569 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:30:00.384545    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df4025b_7993_439a_b856_447d971728ec.slice/crio-46cd7ed4403cc1641ef7ad661971133c8b149cfeed4580dd8f9b5fb0710118f5 WatchSource:0}: Error finding container 46cd7ed4403cc1641ef7ad661971133c8b149cfeed4580dd8f9b5fb0710118f5: Status 404 returned error can't find the container with id 46cd7ed4403cc1641ef7ad661971133c8b149cfeed4580dd8f9b5fb0710118f5
Jul 28 01:30:00.554448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:00.554411    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"]
Jul 28 01:30:00.561601 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:30:00.561571    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e3dd7fa_6317_4dfd_9d86_558913e31a66.slice/crio-d48d62c89435ac591099ff3cb8aa85f8e84437e17f016139a1f8df5fa11362d5 WatchSource:0}: Error finding container d48d62c89435ac591099ff3cb8aa85f8e84437e17f016139a1f8df5fa11362d5: Status 404 returned error can't find the container with id d48d62c89435ac591099ff3cb8aa85f8e84437e17f016139a1f8df5fa11362d5
Jul 28 01:30:01.297292 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.297254    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm" event={"ID":"1e3dd7fa-6317-4dfd-9d86-558913e31a66","Type":"ContainerStarted","Data":"a5344b69f9e98e39abbe3df9f259211de1650c60f0df753cd3d40d240027dfd9"}
Jul 28 01:30:01.297706 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.297324    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm" event={"ID":"1e3dd7fa-6317-4dfd-9d86-558913e31a66","Type":"ContainerStarted","Data":"d48d62c89435ac591099ff3cb8aa85f8e84437e17f016139a1f8df5fa11362d5"}
Jul 28 01:30:01.298745 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.298721    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm" event={"ID":"cb5d3941-844d-4964-866b-366c1ad6d34c","Type":"ContainerStarted","Data":"3cb785e9edbc6ebdc0734ccce93c38c95df81873ded1e16f86b84742887f92cd"}
Jul 28 01:30:01.298846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.298766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm" event={"ID":"cb5d3941-844d-4964-866b-366c1ad6d34c","Type":"ContainerStarted","Data":"a1b0d302ad124d7b98b7a52fd38c7e6d53ff72c3a66437ac0a14fb8c97855ef4"}
Jul 28 01:30:01.300186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.300159    2576 generic.go:356] "Generic (PLEG): container finished" podID="9df4025b-7993-439a-b856-447d971728ec" containerID="c4ce90d24f630546a983050a90822cb3fbb1228be6fa073fd8cfd72402000014" exitCode=0
Jul 28 01:30:01.300284 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.300191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k" event={"ID":"9df4025b-7993-439a-b856-447d971728ec","Type":"ContainerDied","Data":"c4ce90d24f630546a983050a90822cb3fbb1228be6fa073fd8cfd72402000014"}
Jul 28 01:30:01.300284 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.300247    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k" event={"ID":"9df4025b-7993-439a-b856-447d971728ec","Type":"ContainerStarted","Data":"46cd7ed4403cc1641ef7ad661971133c8b149cfeed4580dd8f9b5fb0710118f5"}
Jul 28 01:30:01.312042 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.311998    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm" podStartSLOduration=1.311980527 podStartE2EDuration="1.311980527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:30:01.311963431 +0000 UTC m=+2606.061102156" watchObservedRunningTime="2026-07-28 01:30:01.311980527 +0000 UTC m=+2606.061119252"
Jul 28 01:30:01.339829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:01.339776    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm" podStartSLOduration=1.33976234 podStartE2EDuration="1.33976234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:30:01.339088127 +0000 UTC m=+2606.088226851" watchObservedRunningTime="2026-07-28 01:30:01.33976234 +0000 UTC m=+2606.088901063"
Jul 28 01:30:02.307086 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:02.307051    2576 generic.go:356] "Generic (PLEG): container finished" podID="1e3dd7fa-6317-4dfd-9d86-558913e31a66" containerID="a5344b69f9e98e39abbe3df9f259211de1650c60f0df753cd3d40d240027dfd9" exitCode=0
Jul 28 01:30:02.307535 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:02.307146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm" event={"ID":"1e3dd7fa-6317-4dfd-9d86-558913e31a66","Type":"ContainerDied","Data":"a5344b69f9e98e39abbe3df9f259211de1650c60f0df753cd3d40d240027dfd9"}
Jul 28 01:30:02.308607 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:02.308586    2576 generic.go:356] "Generic (PLEG): container finished" podID="cb5d3941-844d-4964-866b-366c1ad6d34c" containerID="3cb785e9edbc6ebdc0734ccce93c38c95df81873ded1e16f86b84742887f92cd" exitCode=0
Jul 28 01:30:02.308663 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:02.308635    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm" event={"ID":"cb5d3941-844d-4964-866b-366c1ad6d34c","Type":"ContainerDied","Data":"3cb785e9edbc6ebdc0734ccce93c38c95df81873ded1e16f86b84742887f92cd"}
Jul 28 01:30:02.441912 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:02.441886    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"
Jul 28 01:30:03.314854 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:03.314822    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"
Jul 28 01:30:03.315297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:03.314856    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k" event={"ID":"9df4025b-7993-439a-b856-447d971728ec","Type":"ContainerDied","Data":"46cd7ed4403cc1641ef7ad661971133c8b149cfeed4580dd8f9b5fb0710118f5"}
Jul 28 01:30:03.315297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:03.314897    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46cd7ed4403cc1641ef7ad661971133c8b149cfeed4580dd8f9b5fb0710118f5"
Jul 28 01:30:03.476693 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:03.476671    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"
Jul 28 01:30:03.479515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:03.479495    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"
Jul 28 01:30:04.320515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:04.320481    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"
Jul 28 01:30:04.320947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:04.320534    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm" event={"ID":"1e3dd7fa-6317-4dfd-9d86-558913e31a66","Type":"ContainerDied","Data":"d48d62c89435ac591099ff3cb8aa85f8e84437e17f016139a1f8df5fa11362d5"}
Jul 28 01:30:04.320947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:04.320595    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d48d62c89435ac591099ff3cb8aa85f8e84437e17f016139a1f8df5fa11362d5"
Jul 28 01:30:04.321765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:04.321743    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm"
Jul 28 01:30:04.321888 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:04.321761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753370-7fdzm" event={"ID":"cb5d3941-844d-4964-866b-366c1ad6d34c","Type":"ContainerDied","Data":"a1b0d302ad124d7b98b7a52fd38c7e6d53ff72c3a66437ac0a14fb8c97855ef4"}
Jul 28 01:30:04.321888 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:04.321799    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1b0d302ad124d7b98b7a52fd38c7e6d53ff72c3a66437ac0a14fb8c97855ef4"
Jul 28 01:30:41.090614 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:30:41.090582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:41.329959 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:31:41.329925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:35.988894 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:32:35.988860    2576 scope.go:117] "RemoveContainer" containerID="9b0f7fc4550d42dba73040693f8ff27a3e36def22efd8dbb0764695d8a24ebe4"
Jul 28 01:32:49.204760 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:32:49.204728    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"]
Jul 28 01:32:49.232787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:32:49.232755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:50.264478 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:32:50.264444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:01.527883 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.527843    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"]
Jul 28 01:33:01.528288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528265    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e3dd7fa-6317-4dfd-9d86-558913e31a66" containerName="cleanup"
Jul 28 01:33:01.528288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528278    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e3dd7fa-6317-4dfd-9d86-558913e31a66" containerName="cleanup"
Jul 28 01:33:01.528288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528286    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9df4025b-7993-439a-b856-447d971728ec" containerName="cleanup"
Jul 28 01:33:01.528404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528291    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9df4025b-7993-439a-b856-447d971728ec" containerName="cleanup"
Jul 28 01:33:01.528404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528306    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb5d3941-844d-4964-866b-366c1ad6d34c" containerName="cleanup"
Jul 28 01:33:01.528404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528312    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb5d3941-844d-4964-866b-366c1ad6d34c" containerName="cleanup"
Jul 28 01:33:01.528404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528376    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e3dd7fa-6317-4dfd-9d86-558913e31a66" containerName="cleanup"
Jul 28 01:33:01.528404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528385    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9df4025b-7993-439a-b856-447d971728ec" containerName="cleanup"
Jul 28 01:33:01.528404 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.528392    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb5d3941-844d-4964-866b-366c1ad6d34c" containerName="cleanup"
Jul 28 01:33:01.531527 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.531504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.534269 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.534244    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-14a1d0\"/\"sub-test-model-14a1d0-kserve-self-signed-certs\""
Jul 28 01:33:01.543931 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.543903    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"]
Jul 28 01:33:01.610074 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.610035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-model-cache\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.610297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.610099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a18731-8700-4659-b442-f867eead405e-tls-certs\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.610297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.610141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-home\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.610297 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.610167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-tmp-dir\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.610467 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.610306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-dshm\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.610467 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.610356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5wn\" (UniqueName: \"kubernetes.io/projected/04a18731-8700-4659-b442-f867eead405e-kube-api-access-ww5wn\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.613435 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.613409    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:01.711347 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-tmp-dir\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711519 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711358    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-dshm\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711519 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711384    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5wn\" (UniqueName: \"kubernetes.io/projected/04a18731-8700-4659-b442-f867eead405e-kube-api-access-ww5wn\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711519 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-model-cache\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711519 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a18731-8700-4659-b442-f867eead405e-tls-certs\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711519 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-home\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711841 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711809    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-tmp-dir\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711911 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-model-cache\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.711976 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.711959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-home\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.713638 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.713614    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-dshm\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.713834 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.713818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a18731-8700-4659-b442-f867eead405e-tls-certs\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.719491 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.719469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5wn\" (UniqueName: \"kubernetes.io/projected/04a18731-8700-4659-b442-f867eead405e-kube-api-access-ww5wn\") pod \"sub-test-model-14a1d0-kserve-85cb597f7c-h898q\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") " pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.842455 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.842424    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:01.981066 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:01.981043    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"]
Jul 28 01:33:01.982935 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:33:01.982900    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04a18731_8700_4659_b442_f867eead405e.slice/crio-2c6453f07d1101375b16c8fc62b6fc7800f44129d2dc5d552c216adfc9c95dfb WatchSource:0}: Error finding container 2c6453f07d1101375b16c8fc62b6fc7800f44129d2dc5d552c216adfc9c95dfb: Status 404 returned error can't find the container with id 2c6453f07d1101375b16c8fc62b6fc7800f44129d2dc5d552c216adfc9c95dfb
Jul 28 01:33:02.168936 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:02.168848    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q" event={"ID":"04a18731-8700-4659-b442-f867eead405e","Type":"ContainerStarted","Data":"de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1"}
Jul 28 01:33:02.168936 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:02.168894    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q" event={"ID":"04a18731-8700-4659-b442-f867eead405e","Type":"ContainerStarted","Data":"2c6453f07d1101375b16c8fc62b6fc7800f44129d2dc5d552c216adfc9c95dfb"}
Jul 28 01:33:02.186566 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:02.186520    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q" podStartSLOduration=1.186503796 podStartE2EDuration="1.186503796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:02.185764611 +0000 UTC m=+2786.934903349" watchObservedRunningTime="2026-07-28 01:33:02.186503796 +0000 UTC m=+2786.935642520"
Jul 28 01:33:04.223080 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.223043    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b" podUID="4a345026-6ea0-4939-bc1c-cbc2fe599177" containerName="main" containerID="cri-o://32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc" gracePeriod=45
Jul 28 01:33:04.477671 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.477615    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:33:04.638643 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-model-cache\") pod \"4a345026-6ea0-4939-bc1c-cbc2fe599177\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") "
Jul 28 01:33:04.638820 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a345026-6ea0-4939-bc1c-cbc2fe599177-tls-certs\") pod \"4a345026-6ea0-4939-bc1c-cbc2fe599177\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") "
Jul 28 01:33:04.638820 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638693    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-tmp-dir\") pod \"4a345026-6ea0-4939-bc1c-cbc2fe599177\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") "
Jul 28 01:33:04.638820 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638721    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-dshm\") pod \"4a345026-6ea0-4939-bc1c-cbc2fe599177\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") "
Jul 28 01:33:04.638820 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638802    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-home\") pod \"4a345026-6ea0-4939-bc1c-cbc2fe599177\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") "
Jul 28 01:33:04.639047 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638854    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h2tj\" (UniqueName: \"kubernetes.io/projected/4a345026-6ea0-4939-bc1c-cbc2fe599177-kube-api-access-8h2tj\") pod \"4a345026-6ea0-4939-bc1c-cbc2fe599177\" (UID: \"4a345026-6ea0-4939-bc1c-cbc2fe599177\") "
Jul 28 01:33:04.639047 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638852    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-model-cache" (OuterVolumeSpecName: "model-cache") pod "4a345026-6ea0-4939-bc1c-cbc2fe599177" (UID: "4a345026-6ea0-4939-bc1c-cbc2fe599177"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:04.639047 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.638983    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4a345026-6ea0-4939-bc1c-cbc2fe599177" (UID: "4a345026-6ea0-4939-bc1c-cbc2fe599177"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:04.639198 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.639067    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-home" (OuterVolumeSpecName: "home") pod "4a345026-6ea0-4939-bc1c-cbc2fe599177" (UID: "4a345026-6ea0-4939-bc1c-cbc2fe599177"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:04.639198 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.639121    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:33:04.639198 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.639140    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:33:04.640894 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.640875    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-dshm" (OuterVolumeSpecName: "dshm") pod "4a345026-6ea0-4939-bc1c-cbc2fe599177" (UID: "4a345026-6ea0-4939-bc1c-cbc2fe599177"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:33:04.640958 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.640942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a345026-6ea0-4939-bc1c-cbc2fe599177-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4a345026-6ea0-4939-bc1c-cbc2fe599177" (UID: "4a345026-6ea0-4939-bc1c-cbc2fe599177"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:33:04.641047 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.641025    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a345026-6ea0-4939-bc1c-cbc2fe599177-kube-api-access-8h2tj" (OuterVolumeSpecName: "kube-api-access-8h2tj") pod "4a345026-6ea0-4939-bc1c-cbc2fe599177" (UID: "4a345026-6ea0-4939-bc1c-cbc2fe599177"). InnerVolumeSpecName "kube-api-access-8h2tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:33:04.740050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.739989    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h2tj\" (UniqueName: \"kubernetes.io/projected/4a345026-6ea0-4939-bc1c-cbc2fe599177-kube-api-access-8h2tj\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:33:04.740050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.740015    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a345026-6ea0-4939-bc1c-cbc2fe599177-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:33:04.740050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.740025    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:33:04.740050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:04.740032    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a345026-6ea0-4939-bc1c-cbc2fe599177-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:33:05.184106 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.184070    2576 generic.go:356] "Generic (PLEG): container finished" podID="4a345026-6ea0-4939-bc1c-cbc2fe599177" containerID="32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc" exitCode=0
Jul 28 01:33:05.184271 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.184113    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"
Jul 28 01:33:05.184271 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.184157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b" event={"ID":"4a345026-6ea0-4939-bc1c-cbc2fe599177","Type":"ContainerDied","Data":"32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc"}
Jul 28 01:33:05.184271 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.184200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b" event={"ID":"4a345026-6ea0-4939-bc1c-cbc2fe599177","Type":"ContainerDied","Data":"be51f95f59f873efd09a3fbced0ebcb2e24416bc048e36e6a894b47b87716ac9"}
Jul 28 01:33:05.184271 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.184231    2576 scope.go:117] "RemoveContainer" containerID="32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc"
Jul 28 01:33:05.193091 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.193074    2576 scope.go:117] "RemoveContainer" containerID="32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc"
Jul 28 01:33:05.193368 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:33:05.193352    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc\": container with ID starting with 32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc not found: ID does not exist" containerID="32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc"
Jul 28 01:33:05.193423 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.193378    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc"} err="failed to get container status \"32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc\": rpc error: code = NotFound desc = could not find container \"32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc\": container with ID starting with 32de677904fc6ad04bf732b9ef3c910fe98295393c6c6b79176c7c3fe45a90dc not found: ID does not exist"
Jul 28 01:33:05.206985 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.206961    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"]
Jul 28 01:33:05.210215 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.210194    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/auth-test-model-14a1d0-kserve-7c54ff9bc5-nbl2b"]
Jul 28 01:33:05.752203 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:05.752166    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a345026-6ea0-4939-bc1c-cbc2fe599177" path="/var/lib/kubelet/pods/4a345026-6ea0-4939-bc1c-cbc2fe599177/volumes"
Jul 28 01:33:11.842891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:11.842857    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:11.843514 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:11.842908    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:11.855060 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:11.855036    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:12.228884 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:12.228814    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:33:27.517829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.517798    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"]
Jul 28 01:33:27.518351 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.518329    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a345026-6ea0-4939-bc1c-cbc2fe599177" containerName="main"
Jul 28 01:33:27.518528 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.518511    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a345026-6ea0-4939-bc1c-cbc2fe599177" containerName="main"
Jul 28 01:33:27.518753 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.518722    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a345026-6ea0-4939-bc1c-cbc2fe599177" containerName="main"
Jul 28 01:33:27.521782 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.521763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.524327 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.524308    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"openshift-service-ca.crt\""
Jul 28 01:33:27.524409 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.524375    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"default-dockercfg-n8zf4\""
Jul 28 01:33:27.525474 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.525453    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"kube-root-ca.crt\""
Jul 28 01:33:27.525551 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.525520    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"sub-test-model-3131e3-kserve-self-signed-certs\""
Jul 28 01:33:27.530598 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.530573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"]
Jul 28 01:33:27.561324 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.561298    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:27.637143 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.637113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-home\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.637284 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.637152    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-dshm\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.637284 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.637180    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tls-certs\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.637284 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.637198    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tmp-dir\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.637479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.637278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdzz\" (UniqueName: \"kubernetes.io/projected/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-kube-api-access-dqdzz\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.637479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.637315    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-model-cache\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.737887 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.737862    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-dshm\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738007 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.737901    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tls-certs\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738007 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.737922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tmp-dir\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738007 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.737952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdzz\" (UniqueName: \"kubernetes.io/projected/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-kube-api-access-dqdzz\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738007 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.737977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-model-cache\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738263 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.738148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-home\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738360 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.738340    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tmp-dir\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738432 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.738389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-model-cache\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.738571 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.738552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-home\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.740713 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.740691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-dshm\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.744267 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.741348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tls-certs\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.750449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.750429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdzz\" (UniqueName: \"kubernetes.io/projected/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-kube-api-access-dqdzz\") pod \"sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") " pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.831808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.831782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:27.971199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:27.971174    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"]
Jul 28 01:33:27.972914 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:33:27.972886    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b0a7e7b_b0dd_41ae_9e18_0461d67bcfcc.slice/crio-139fdfa8b1df33383bcf7af6430007c0a7bb0ac3393483c5835feac6b8f0e0fb WatchSource:0}: Error finding container 139fdfa8b1df33383bcf7af6430007c0a7bb0ac3393483c5835feac6b8f0e0fb: Status 404 returned error can't find the container with id 139fdfa8b1df33383bcf7af6430007c0a7bb0ac3393483c5835feac6b8f0e0fb
Jul 28 01:33:28.295853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:28.295768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh" event={"ID":"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc","Type":"ContainerStarted","Data":"7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f"}
Jul 28 01:33:28.295853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:28.295812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh" event={"ID":"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc","Type":"ContainerStarted","Data":"139fdfa8b1df33383bcf7af6430007c0a7bb0ac3393483c5835feac6b8f0e0fb"}
Jul 28 01:33:28.312731 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:28.312682    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh" podStartSLOduration=1.312666845 podStartE2EDuration="1.312666845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:33:28.311265548 +0000 UTC m=+2813.060404274" watchObservedRunningTime="2026-07-28 01:33:28.312666845 +0000 UTC m=+2813.061805568"
Jul 28 01:33:37.832515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:37.832482    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:37.832901 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:37.832529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:37.845204 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:37.845181    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:38.351955 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:38.351927    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:33:42.749377 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:42.749342    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:51.098191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:51.098137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:51.167435 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:33:51.167400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:47.426934 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:34:47.426897    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"]
Jul 28 01:34:47.455753 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:34:47.455720    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:48.228574 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:34:48.228543    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"]
Jul 28 01:34:48.460296 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:34:48.460264    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:56.634016 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:34:56.633979    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:02.451479 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.451443    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q" podUID="04a18731-8700-4659-b442-f867eead405e" containerName="main" containerID="cri-o://de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1" gracePeriod=45
Jul 28 01:35:02.692779 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.692753    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:35:02.738672 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.738606    2576 generic.go:356] "Generic (PLEG): container finished" podID="04a18731-8700-4659-b442-f867eead405e" containerID="de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1" exitCode=0
Jul 28 01:35:02.738672 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.738637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q" event={"ID":"04a18731-8700-4659-b442-f867eead405e","Type":"ContainerDied","Data":"de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1"}
Jul 28 01:35:02.738672 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.738661    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"
Jul 28 01:35:02.738874 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.738681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q" event={"ID":"04a18731-8700-4659-b442-f867eead405e","Type":"ContainerDied","Data":"2c6453f07d1101375b16c8fc62b6fc7800f44129d2dc5d552c216adfc9c95dfb"}
Jul 28 01:35:02.738874 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.738699    2576 scope.go:117] "RemoveContainer" containerID="de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1"
Jul 28 01:35:02.748441 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.748423    2576 scope.go:117] "RemoveContainer" containerID="de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1"
Jul 28 01:35:02.748698 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:35:02.748671    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1\": container with ID starting with de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1 not found: ID does not exist" containerID="de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1"
Jul 28 01:35:02.748796 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.748707    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1"} err="failed to get container status \"de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1\": rpc error: code = NotFound desc = could not find container \"de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1\": container with ID starting with de15f039952a25e0d3c57b9ae4d5e4b15bc59825d8defb7129ec1ed7cfdba3e1 not found: ID does not exist"
Jul 28 01:35:02.771950 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.771928    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-home\") pod \"04a18731-8700-4659-b442-f867eead405e\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") "
Jul 28 01:35:02.772041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.771960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-tmp-dir\") pod \"04a18731-8700-4659-b442-f867eead405e\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") "
Jul 28 01:35:02.772041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.771975    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-dshm\") pod \"04a18731-8700-4659-b442-f867eead405e\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") "
Jul 28 01:35:02.772041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772004    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-model-cache\") pod \"04a18731-8700-4659-b442-f867eead405e\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") "
Jul 28 01:35:02.772162 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772065    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5wn\" (UniqueName: \"kubernetes.io/projected/04a18731-8700-4659-b442-f867eead405e-kube-api-access-ww5wn\") pod \"04a18731-8700-4659-b442-f867eead405e\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") "
Jul 28 01:35:02.772208 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a18731-8700-4659-b442-f867eead405e-tls-certs\") pod \"04a18731-8700-4659-b442-f867eead405e\" (UID: \"04a18731-8700-4659-b442-f867eead405e\") "
Jul 28 01:35:02.772299 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772241    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04a18731-8700-4659-b442-f867eead405e" (UID: "04a18731-8700-4659-b442-f867eead405e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:02.772341 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772248    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-home" (OuterVolumeSpecName: "home") pod "04a18731-8700-4659-b442-f867eead405e" (UID: "04a18731-8700-4659-b442-f867eead405e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:02.772341 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772321    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-model-cache" (OuterVolumeSpecName: "model-cache") pod "04a18731-8700-4659-b442-f867eead405e" (UID: "04a18731-8700-4659-b442-f867eead405e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:02.772535 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772515    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:02.772655 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772539    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:02.772655 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.772553    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:02.774100 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.774077    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a18731-8700-4659-b442-f867eead405e-kube-api-access-ww5wn" (OuterVolumeSpecName: "kube-api-access-ww5wn") pod "04a18731-8700-4659-b442-f867eead405e" (UID: "04a18731-8700-4659-b442-f867eead405e"). InnerVolumeSpecName "kube-api-access-ww5wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:35:02.774490 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.774463    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-dshm" (OuterVolumeSpecName: "dshm") pod "04a18731-8700-4659-b442-f867eead405e" (UID: "04a18731-8700-4659-b442-f867eead405e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:02.774490 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.774463    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a18731-8700-4659-b442-f867eead405e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04a18731-8700-4659-b442-f867eead405e" (UID: "04a18731-8700-4659-b442-f867eead405e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:35:02.873825 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.873799    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5wn\" (UniqueName: \"kubernetes.io/projected/04a18731-8700-4659-b442-f867eead405e-kube-api-access-ww5wn\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:02.873825 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.873823    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04a18731-8700-4659-b442-f867eead405e-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:02.873962 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:02.873833    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04a18731-8700-4659-b442-f867eead405e-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.064062 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.064028    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"]
Jul 28 01:35:03.067676 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.067643    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-14a1d0/sub-test-model-14a1d0-kserve-85cb597f7c-h898q"]
Jul 28 01:35:03.247385 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.247342    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh" podUID="0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" containerName="main" containerID="cri-o://7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f" gracePeriod=45
Jul 28 01:35:03.490681 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.490657    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:35:03.578913 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.578884    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-home\") pod \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") "
Jul 28 01:35:03.579072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.578921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-model-cache\") pod \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") "
Jul 28 01:35:03.579072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.578937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-dshm\") pod \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") "
Jul 28 01:35:03.579072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.578966    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqdzz\" (UniqueName: \"kubernetes.io/projected/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-kube-api-access-dqdzz\") pod \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") "
Jul 28 01:35:03.579072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.578999    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tls-certs\") pod \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") "
Jul 28 01:35:03.579072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.579013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tmp-dir\") pod \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\" (UID: \"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc\") "
Jul 28 01:35:03.579359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.579154    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" (UID: "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:03.579359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.579203    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-home" (OuterVolumeSpecName: "home") pod "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" (UID: "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:03.579462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.579361    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.579462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.579382    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.579462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.579396    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" (UID: "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:03.581041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.581014    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-dshm" (OuterVolumeSpecName: "dshm") pod "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" (UID: "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:03.581167 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.581148    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" (UID: "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:35:03.581213 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.581173    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-kube-api-access-dqdzz" (OuterVolumeSpecName: "kube-api-access-dqdzz") pod "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" (UID: "0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc"). InnerVolumeSpecName "kube-api-access-dqdzz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:35:03.679987 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.679929    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqdzz\" (UniqueName: \"kubernetes.io/projected/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-kube-api-access-dqdzz\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.679987 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.679950    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.679987 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.679979    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.679987 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.679987    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:35:03.745723 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.745692    2576 generic.go:356] "Generic (PLEG): container finished" podID="0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" containerID="7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f" exitCode=0
Jul 28 01:35:03.745836 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.745809    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"
Jul 28 01:35:03.751384 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.751361    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a18731-8700-4659-b442-f867eead405e" path="/var/lib/kubelet/pods/04a18731-8700-4659-b442-f867eead405e/volumes"
Jul 28 01:35:03.751738 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.751722    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh" event={"ID":"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc","Type":"ContainerDied","Data":"7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f"}
Jul 28 01:35:03.751798 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.751748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh" event={"ID":"0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc","Type":"ContainerDied","Data":"139fdfa8b1df33383bcf7af6430007c0a7bb0ac3393483c5835feac6b8f0e0fb"}
Jul 28 01:35:03.751798 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.751767    2576 scope.go:117] "RemoveContainer" containerID="7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f"
Jul 28 01:35:03.761267 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.761253    2576 scope.go:117] "RemoveContainer" containerID="7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f"
Jul 28 01:35:03.761532 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:35:03.761516    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f\": container with ID starting with 7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f not found: ID does not exist" containerID="7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f"
Jul 28 01:35:03.761571 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.761541    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f"} err="failed to get container status \"7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f\": rpc error: code = NotFound desc = could not find container \"7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f\": container with ID starting with 7ebaf006b584463f66d09d6f62babb5eab3e26f7c9467792a5d03ff2a3b3f01f not found: ID does not exist"
Jul 28 01:35:03.776500 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.776479    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"]
Jul 28 01:35:03.780015 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:03.779993    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/sub-test-model-3131e3-kserve-5f48bcbb59-tn4dh"]
Jul 28 01:35:05.751215 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:05.751174    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" path="/var/lib/kubelet/pods/0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc/volumes"
Jul 28 01:35:23.635910 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.635877    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"]
Jul 28 01:35:23.636421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.636302    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a18731-8700-4659-b442-f867eead405e" containerName="main"
Jul 28 01:35:23.636421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.636316    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a18731-8700-4659-b442-f867eead405e" containerName="main"
Jul 28 01:35:23.636421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.636342    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" containerName="main"
Jul 28 01:35:23.636421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.636348    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" containerName="main"
Jul 28 01:35:23.636421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.636408    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a18731-8700-4659-b442-f867eead405e" containerName="main"
Jul 28 01:35:23.636421 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.636415    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0a7e7b-b0dd-41ae-9e18-0461d67bcfcc" containerName="main"
Jul 28 01:35:23.639465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.639450    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.643163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.643139    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"rate-test-model-3131e3-kserve-self-signed-certs\""
Jul 28 01:35:23.643163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.643149    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"openshift-service-ca.crt\""
Jul 28 01:35:23.643163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.643150    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"default-dockercfg-n8zf4\""
Jul 28 01:35:23.643425 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.643241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-3131e3\"/\"kube-root-ca.crt\""
Jul 28 01:35:23.649050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.649029    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"]
Jul 28 01:35:23.659763 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.659743    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:23.752490 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.752463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-model-cache\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.752593 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.752497    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-dshm\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.752593 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.752523    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-home\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.752593 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.752549    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-tmp-dir\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.752593 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.752574    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f16ce-2a73-478f-abd3-7cce8763d9ab-tls-certs\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.752593 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.752590    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsr7l\" (UniqueName: \"kubernetes.io/projected/847f16ce-2a73-478f-abd3-7cce8763d9ab-kube-api-access-vsr7l\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.853371 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f16ce-2a73-478f-abd3-7cce8763d9ab-tls-certs\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.853481 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsr7l\" (UniqueName: \"kubernetes.io/projected/847f16ce-2a73-478f-abd3-7cce8763d9ab-kube-api-access-vsr7l\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.853481 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-model-cache\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.853481 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-dshm\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.853633 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853605    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-home\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.853687 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-tmp-dir\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.854002 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-model-cache\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.854002 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.853995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-home\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.854180 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.854048    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-tmp-dir\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.855989 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.855966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f16ce-2a73-478f-abd3-7cce8763d9ab-tls-certs\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.856073 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.855970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-dshm\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.861832 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.861814    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsr7l\" (UniqueName: \"kubernetes.io/projected/847f16ce-2a73-478f-abd3-7cce8763d9ab-kube-api-access-vsr7l\") pod \"rate-test-model-3131e3-kserve-586958c8cf-zjspk\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") " pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:23.948941 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:23.948869    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:24.082053 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:24.082005    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"]
Jul 28 01:35:24.084131 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:35:24.084100    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod847f16ce_2a73_478f_abd3_7cce8763d9ab.slice/crio-0994ec091588d949c49831aad9ad9d5aeace0536b49f9945b5426bf97aa52e4a WatchSource:0}: Error finding container 0994ec091588d949c49831aad9ad9d5aeace0536b49f9945b5426bf97aa52e4a: Status 404 returned error can't find the container with id 0994ec091588d949c49831aad9ad9d5aeace0536b49f9945b5426bf97aa52e4a
Jul 28 01:35:24.841793 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:24.841753    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk" event={"ID":"847f16ce-2a73-478f-abd3-7cce8763d9ab","Type":"ContainerStarted","Data":"c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8"}
Jul 28 01:35:24.842197 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:24.841803    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk" event={"ID":"847f16ce-2a73-478f-abd3-7cce8763d9ab","Type":"ContainerStarted","Data":"0994ec091588d949c49831aad9ad9d5aeace0536b49f9945b5426bf97aa52e4a"}
Jul 28 01:35:24.860612 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:24.860563    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk" podStartSLOduration=1.860547377 podStartE2EDuration="1.860547377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:35:24.860490631 +0000 UTC m=+2929.609629355" watchObservedRunningTime="2026-07-28 01:35:24.860547377 +0000 UTC m=+2929.609686090"
Jul 28 01:35:33.949746 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:33.949710    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:33.950267 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:33.949760    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:33.962738 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:33.962711    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:35:34.905159 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:35:34.905130    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:36:36.431382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:36.431315    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:37.329064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:37.329032    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"]
Jul 28 01:36:37.352416 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:37.352377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:38.297449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.297415    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-139833220191920"]
Jul 28 01:36:38.300809 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.300791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:38.303315 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.303289    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-wh7sz\""
Jul 28 01:36:38.305341 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.305315    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833220191920"]
Jul 28 01:36:38.486094 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.486065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56ltn\" (UniqueName: \"kubernetes.io/projected/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba-kube-api-access-56ltn\") pod \"test-curl-7478-139833220191920\" (UID: \"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba\") " pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:38.587451 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.587420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56ltn\" (UniqueName: \"kubernetes.io/projected/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba-kube-api-access-56ltn\") pod \"test-curl-7478-139833220191920\" (UID: \"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba\") " pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:38.595551 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.595530    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56ltn\" (UniqueName: \"kubernetes.io/projected/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba-kube-api-access-56ltn\") pod \"test-curl-7478-139833220191920\" (UID: \"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba\") " pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:38.612026 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.611561    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:38.738872 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.738843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833220191920"]
Jul 28 01:36:38.741005 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:36:38.740980    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ba186d3_457b_4dbc_a5f3_c0be8deeb0ba.slice/crio-7eb799428288e4b39e1c045b6f505070cca8b4fb468eeef04f9975f5b09e84e5 WatchSource:0}: Error finding container 7eb799428288e4b39e1c045b6f505070cca8b4fb468eeef04f9975f5b09e84e5: Status 404 returned error can't find the container with id 7eb799428288e4b39e1c045b6f505070cca8b4fb468eeef04f9975f5b09e84e5
Jul 28 01:36:38.742814 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:38.742793    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:36:39.200179 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:39.200146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833220191920" event={"ID":"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba","Type":"ContainerStarted","Data":"7eb799428288e4b39e1c045b6f505070cca8b4fb468eeef04f9975f5b09e84e5"}
Jul 28 01:36:40.206358 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:40.206323    2576 generic.go:356] "Generic (PLEG): container finished" podID="4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba" containerID="ff366e6cfda95590bb6a913413a9ea8cc8a6bf91b3fb056f5e58ea1c111b399c" exitCode=0
Jul 28 01:36:40.206805 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:40.206391    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833220191920" event={"ID":"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba","Type":"ContainerDied","Data":"ff366e6cfda95590bb6a913413a9ea8cc8a6bf91b3fb056f5e58ea1c111b399c"}
Jul 28 01:36:41.345211 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.345181    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:41.366542 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.366516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7478-139833220191920_4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba/test-curl-7478-139833220191920/0.log"
Jul 28 01:36:41.390932 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.390904    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-139833220191920"]
Jul 28 01:36:41.396229 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.396194    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-139833220191920"]
Jul 28 01:36:41.512414 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.512343    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56ltn\" (UniqueName: \"kubernetes.io/projected/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba-kube-api-access-56ltn\") pod \"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba\" (UID: \"4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba\") "
Jul 28 01:36:41.513143 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.513119    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-139833219838256"]
Jul 28 01:36:41.513510 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.513495    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba" containerName="test-curl-7478-139833220191920"
Jul 28 01:36:41.513510 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.513510    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba" containerName="test-curl-7478-139833220191920"
Jul 28 01:36:41.513653 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.513578    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba" containerName="test-curl-7478-139833220191920"
Jul 28 01:36:41.514595 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.514574    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba-kube-api-access-56ltn" (OuterVolumeSpecName: "kube-api-access-56ltn") pod "4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba" (UID: "4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba"). InnerVolumeSpecName "kube-api-access-56ltn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:41.516610 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.516594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:41.520622 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.520490    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219838256"]
Jul 28 01:36:41.613493 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.613463    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56ltn\" (UniqueName: \"kubernetes.io/projected/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba-kube-api-access-56ltn\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:41.714698 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.714670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r44m8\" (UniqueName: \"kubernetes.io/projected/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa-kube-api-access-r44m8\") pod \"test-curl-7478-139833219838256\" (UID: \"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa\") " pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:41.751417 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.751394    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba" path="/var/lib/kubelet/pods/4ba186d3-457b-4dbc-a5f3-c0be8deeb0ba/volumes"
Jul 28 01:36:41.815322 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.815303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r44m8\" (UniqueName: \"kubernetes.io/projected/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa-kube-api-access-r44m8\") pod \"test-curl-7478-139833219838256\" (UID: \"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa\") " pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:41.824765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.824744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r44m8\" (UniqueName: \"kubernetes.io/projected/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa-kube-api-access-r44m8\") pod \"test-curl-7478-139833219838256\" (UID: \"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa\") " pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:41.830604 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.830585    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:41.958146 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:41.958110    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219838256"]
Jul 28 01:36:41.959627 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:36:41.959596    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb2aafce_6ae6_4317_b8ef_b05ddd7d1daa.slice/crio-d0658426fdd4472396a4204cdb4765133e382ec0a74f36b7d89eda78a38b06fc WatchSource:0}: Error finding container d0658426fdd4472396a4204cdb4765133e382ec0a74f36b7d89eda78a38b06fc: Status 404 returned error can't find the container with id d0658426fdd4472396a4204cdb4765133e382ec0a74f36b7d89eda78a38b06fc
Jul 28 01:36:42.216433 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:42.216360    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833220191920"
Jul 28 01:36:42.216433 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:42.216379    2576 scope.go:117] "RemoveContainer" containerID="ff366e6cfda95590bb6a913413a9ea8cc8a6bf91b3fb056f5e58ea1c111b399c"
Jul 28 01:36:42.217750 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:42.217667    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219838256" event={"ID":"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa","Type":"ContainerStarted","Data":"d0658426fdd4472396a4204cdb4765133e382ec0a74f36b7d89eda78a38b06fc"}
Jul 28 01:36:43.224461 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:43.224426    2576 generic.go:356] "Generic (PLEG): container finished" podID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" containerID="31b5b640ca6868596db624c895f6e577117f5216be50ebfc0a07c7ed09e30722" exitCode=0
Jul 28 01:36:43.224830 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:43.224498    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219838256" event={"ID":"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa","Type":"ContainerDied","Data":"31b5b640ca6868596db624c895f6e577117f5216be50ebfc0a07c7ed09e30722"}
Jul 28 01:36:44.368285 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.368260    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:44.391394 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.391370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7478-139833219838256_bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa/test-curl-7478-139833219838256/0.log"
Jul 28 01:36:44.414685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.414657    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-139833219838256"]
Jul 28 01:36:44.418445 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.418422    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-139833219838256"]
Jul 28 01:36:44.436920 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.436903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r44m8\" (UniqueName: \"kubernetes.io/projected/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa-kube-api-access-r44m8\") pod \"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa\" (UID: \"bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa\") "
Jul 28 01:36:44.438838 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.438816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa-kube-api-access-r44m8" (OuterVolumeSpecName: "kube-api-access-r44m8") pod "bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" (UID: "bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa"). InnerVolumeSpecName "kube-api-access-r44m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:44.537732 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.537673    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r44m8\" (UniqueName: \"kubernetes.io/projected/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa-kube-api-access-r44m8\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:44.572604 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.572583    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:44.572961 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.572949    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" containerName="test-curl-7478-139833219838256"
Jul 28 01:36:44.573009 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.572963    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" containerName="test-curl-7478-139833219838256"
Jul 28 01:36:44.573044 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.573031    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" containerName="test-curl-7478-139833219838256"
Jul 28 01:36:44.575963 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.575949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:44.579760 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.579741    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:44.638841 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.638817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtfq5\" (UniqueName: \"kubernetes.io/projected/c5bc9922-73e0-4596-997d-df5afee8c1c9-kube-api-access-mtfq5\") pod \"test-curl-7478-139833219895488\" (UID: \"c5bc9922-73e0-4596-997d-df5afee8c1c9\") " pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:44.739565 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.739534    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtfq5\" (UniqueName: \"kubernetes.io/projected/c5bc9922-73e0-4596-997d-df5afee8c1c9-kube-api-access-mtfq5\") pod \"test-curl-7478-139833219895488\" (UID: \"c5bc9922-73e0-4596-997d-df5afee8c1c9\") " pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:44.748233 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.748207    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtfq5\" (UniqueName: \"kubernetes.io/projected/c5bc9922-73e0-4596-997d-df5afee8c1c9-kube-api-access-mtfq5\") pod \"test-curl-7478-139833219895488\" (UID: \"c5bc9922-73e0-4596-997d-df5afee8c1c9\") " pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:44.885829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:44.885802    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:45.010412 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:45.010388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:45.013175 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:36:45.013146    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5bc9922_73e0_4596_997d_df5afee8c1c9.slice/crio-4f1e7662689986fb7945125b9cfd53161cf67d318f3343a8e3f1c98a1d6bf006 WatchSource:0}: Error finding container 4f1e7662689986fb7945125b9cfd53161cf67d318f3343a8e3f1c98a1d6bf006: Status 404 returned error can't find the container with id 4f1e7662689986fb7945125b9cfd53161cf67d318f3343a8e3f1c98a1d6bf006
Jul 28 01:36:45.237734 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:45.237664    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:45.237734 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:45.237694    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0658426fdd4472396a4204cdb4765133e382ec0a74f36b7d89eda78a38b06fc"
Jul 28 01:36:45.238855 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:45.238832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219895488" event={"ID":"c5bc9922-73e0-4596-997d-df5afee8c1c9","Type":"ContainerStarted","Data":"4f1e7662689986fb7945125b9cfd53161cf67d318f3343a8e3f1c98a1d6bf006"}
Jul 28 01:36:45.249835 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:36:45.249810    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-139833219838256\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:45.749407 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:36:45.749363    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-139833219838256\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" pod="opendatahub/test-curl-7478-139833219838256"
Jul 28 01:36:45.750705 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:45.750683    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa" path="/var/lib/kubelet/pods/bb2aafce-6ae6-4317-b8ef-b05ddd7d1daa/volumes"
Jul 28 01:36:46.244498 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:46.244465    2576 generic.go:356] "Generic (PLEG): container finished" podID="c5bc9922-73e0-4596-997d-df5afee8c1c9" containerID="c52994a4d4f052f4e1d367677b8dd0eb9440259341f6fbe6d0bb2fc8d816516e" exitCode=0
Jul 28 01:36:46.244668 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:46.244552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219895488" event={"ID":"c5bc9922-73e0-4596-997d-df5afee8c1c9","Type":"ContainerDied","Data":"c52994a4d4f052f4e1d367677b8dd0eb9440259341f6fbe6d0bb2fc8d816516e"}
Jul 28 01:36:47.393438 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.393414    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:47.412938 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.412870    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7478-139833219895488_c5bc9922-73e0-4596-997d-df5afee8c1c9/test-curl-7478-139833219895488/0.log"
Jul 28 01:36:47.436191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.436160    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:47.439320 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.439297    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:47.460906 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.460879    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtfq5\" (UniqueName: \"kubernetes.io/projected/c5bc9922-73e0-4596-997d-df5afee8c1c9-kube-api-access-mtfq5\") pod \"c5bc9922-73e0-4596-997d-df5afee8c1c9\" (UID: \"c5bc9922-73e0-4596-997d-df5afee8c1c9\") "
Jul 28 01:36:47.462991 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.462965    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bc9922-73e0-4596-997d-df5afee8c1c9-kube-api-access-mtfq5" (OuterVolumeSpecName: "kube-api-access-mtfq5") pod "c5bc9922-73e0-4596-997d-df5afee8c1c9" (UID: "c5bc9922-73e0-4596-997d-df5afee8c1c9"). InnerVolumeSpecName "kube-api-access-mtfq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:47.553195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.553127    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:47.553512 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.553498    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5bc9922-73e0-4596-997d-df5afee8c1c9" containerName="test-curl-7478-139833219895488"
Jul 28 01:36:47.553573 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.553514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5bc9922-73e0-4596-997d-df5afee8c1c9" containerName="test-curl-7478-139833219895488"
Jul 28 01:36:47.553615 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.553582    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5bc9922-73e0-4596-997d-df5afee8c1c9" containerName="test-curl-7478-139833219895488"
Jul 28 01:36:47.556580 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.556558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:47.560750 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.560730    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:47.561931 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.561907    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtfq5\" (UniqueName: \"kubernetes.io/projected/c5bc9922-73e0-4596-997d-df5afee8c1c9-kube-api-access-mtfq5\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:47.662892 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.662858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9znjt\" (UniqueName: \"kubernetes.io/projected/c79af279-3194-449d-9b6a-c12a6d5104f0-kube-api-access-9znjt\") pod \"test-curl-7478-139833219895488\" (UID: \"c79af279-3194-449d-9b6a-c12a6d5104f0\") " pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:47.752129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.752097    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5bc9922-73e0-4596-997d-df5afee8c1c9" path="/var/lib/kubelet/pods/c5bc9922-73e0-4596-997d-df5afee8c1c9/volumes"
Jul 28 01:36:47.764384 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.764361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9znjt\" (UniqueName: \"kubernetes.io/projected/c79af279-3194-449d-9b6a-c12a6d5104f0-kube-api-access-9znjt\") pod \"test-curl-7478-139833219895488\" (UID: \"c79af279-3194-449d-9b6a-c12a6d5104f0\") " pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:47.772596 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.772577    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9znjt\" (UniqueName: \"kubernetes.io/projected/c79af279-3194-449d-9b6a-c12a6d5104f0-kube-api-access-9znjt\") pod \"test-curl-7478-139833219895488\" (UID: \"c79af279-3194-449d-9b6a-c12a6d5104f0\") " pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:47.868521 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:47.868489    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:48.011359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:48.011180    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:48.013880 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:36:48.013849    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc79af279_3194_449d_9b6a_c12a6d5104f0.slice/crio-f0d86f28d10b2311f0b8d3b6e1c52395b63247c796cf8cab67a601d0d9703370 WatchSource:0}: Error finding container f0d86f28d10b2311f0b8d3b6e1c52395b63247c796cf8cab67a601d0d9703370: Status 404 returned error can't find the container with id f0d86f28d10b2311f0b8d3b6e1c52395b63247c796cf8cab67a601d0d9703370
Jul 28 01:36:48.256348 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:48.256270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219895488" event={"ID":"c79af279-3194-449d-9b6a-c12a6d5104f0","Type":"ContainerStarted","Data":"f0d86f28d10b2311f0b8d3b6e1c52395b63247c796cf8cab67a601d0d9703370"}
Jul 28 01:36:48.257448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:48.257428    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:48.257448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:48.257443    2576 scope.go:117] "RemoveContainer" containerID="c52994a4d4f052f4e1d367677b8dd0eb9440259341f6fbe6d0bb2fc8d816516e"
Jul 28 01:36:49.264762 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:49.264727    2576 generic.go:356] "Generic (PLEG): container finished" podID="c79af279-3194-449d-9b6a-c12a6d5104f0" containerID="82a1eb539ed83ec9f9aa8e559d135cc45a964ae23009d6843ad0acdfef314bb9" exitCode=0
Jul 28 01:36:49.265124 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:49.264767    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219895488" event={"ID":"c79af279-3194-449d-9b6a-c12a6d5104f0","Type":"ContainerDied","Data":"82a1eb539ed83ec9f9aa8e559d135cc45a964ae23009d6843ad0acdfef314bb9"}
Jul 28 01:36:50.404961 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.404936    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:50.424072 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.424045    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7478-139833219895488_c79af279-3194-449d-9b6a-c12a6d5104f0/test-curl-7478-139833219895488/0.log"
Jul 28 01:36:50.447975 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.447947    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:50.451529 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.451506    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-139833219895488"]
Jul 28 01:36:50.487842 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.487821    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9znjt\" (UniqueName: \"kubernetes.io/projected/c79af279-3194-449d-9b6a-c12a6d5104f0-kube-api-access-9znjt\") pod \"c79af279-3194-449d-9b6a-c12a6d5104f0\" (UID: \"c79af279-3194-449d-9b6a-c12a6d5104f0\") "
Jul 28 01:36:50.489752 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.489732    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c79af279-3194-449d-9b6a-c12a6d5104f0-kube-api-access-9znjt" (OuterVolumeSpecName: "kube-api-access-9znjt") pod "c79af279-3194-449d-9b6a-c12a6d5104f0" (UID: "c79af279-3194-449d-9b6a-c12a6d5104f0"). InnerVolumeSpecName "kube-api-access-9znjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:50.568021 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.567999    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:50.568370 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.568358    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c79af279-3194-449d-9b6a-c12a6d5104f0" containerName="test-curl-7478-139833219895488"
Jul 28 01:36:50.568418 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.568372    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c79af279-3194-449d-9b6a-c12a6d5104f0" containerName="test-curl-7478-139833219895488"
Jul 28 01:36:50.568454 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.568439    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c79af279-3194-449d-9b6a-c12a6d5104f0" containerName="test-curl-7478-139833219895488"
Jul 28 01:36:50.571366 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.571348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:50.579311 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.579293    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:50.589055 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.589035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t95q\" (UniqueName: \"kubernetes.io/projected/43d8c6cd-e28d-4426-86c4-62d4faea3030-kube-api-access-4t95q\") pod \"test-curl-7478-139833219896496\" (UID: \"43d8c6cd-e28d-4426-86c4-62d4faea3030\") " pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:50.589150 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.589139    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9znjt\" (UniqueName: \"kubernetes.io/projected/c79af279-3194-449d-9b6a-c12a6d5104f0-kube-api-access-9znjt\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:50.689730 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.689707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t95q\" (UniqueName: \"kubernetes.io/projected/43d8c6cd-e28d-4426-86c4-62d4faea3030-kube-api-access-4t95q\") pod \"test-curl-7478-139833219896496\" (UID: \"43d8c6cd-e28d-4426-86c4-62d4faea3030\") " pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:50.698100 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.698079    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t95q\" (UniqueName: \"kubernetes.io/projected/43d8c6cd-e28d-4426-86c4-62d4faea3030-kube-api-access-4t95q\") pod \"test-curl-7478-139833219896496\" (UID: \"43d8c6cd-e28d-4426-86c4-62d4faea3030\") " pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:50.881022 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:50.880946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:51.005655 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:51.005619    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:51.005851 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:36:51.005829    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43d8c6cd_e28d_4426_86c4_62d4faea3030.slice/crio-17f2995dfe414c8c2384c6104e3999243b3558504fa065b3bfb91ca71c6cc99e WatchSource:0}: Error finding container 17f2995dfe414c8c2384c6104e3999243b3558504fa065b3bfb91ca71c6cc99e: Status 404 returned error can't find the container with id 17f2995dfe414c8c2384c6104e3999243b3558504fa065b3bfb91ca71c6cc99e
Jul 28 01:36:51.275910 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:51.275882    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:51.276019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:51.275928    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d86f28d10b2311f0b8d3b6e1c52395b63247c796cf8cab67a601d0d9703370"
Jul 28 01:36:51.277063 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:51.277040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219896496" event={"ID":"43d8c6cd-e28d-4426-86c4-62d4faea3030","Type":"ContainerStarted","Data":"17f2995dfe414c8c2384c6104e3999243b3558504fa065b3bfb91ca71c6cc99e"}
Jul 28 01:36:51.287574 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:36:51.287547    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-139833219895488\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="c79af279-3194-449d-9b6a-c12a6d5104f0" pod="opendatahub/test-curl-7478-139833219895488"
Jul 28 01:36:51.752868 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:51.752831    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c79af279-3194-449d-9b6a-c12a6d5104f0" path="/var/lib/kubelet/pods/c79af279-3194-449d-9b6a-c12a6d5104f0/volumes"
Jul 28 01:36:52.282801 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.282763    2576 generic.go:356] "Generic (PLEG): container finished" podID="43d8c6cd-e28d-4426-86c4-62d4faea3030" containerID="91a40c9614c82fa2e6593f969bd29f5d86c7a1308fef81efdb01b47ee0bc79c4" exitCode=0
Jul 28 01:36:52.282976 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.282806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219896496" event={"ID":"43d8c6cd-e28d-4426-86c4-62d4faea3030","Type":"ContainerDied","Data":"91a40c9614c82fa2e6593f969bd29f5d86c7a1308fef81efdb01b47ee0bc79c4"}
Jul 28 01:36:52.348032 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.347994    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk" podUID="847f16ce-2a73-478f-abd3-7cce8763d9ab" containerName="main" containerID="cri-o://c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8" gracePeriod=45
Jul 28 01:36:52.601121 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.601099    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:36:52.709022 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.708995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-dshm\") pod \"847f16ce-2a73-478f-abd3-7cce8763d9ab\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") "
Jul 28 01:36:52.709191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709046    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-model-cache\") pod \"847f16ce-2a73-478f-abd3-7cce8763d9ab\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") "
Jul 28 01:36:52.709191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709070    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsr7l\" (UniqueName: \"kubernetes.io/projected/847f16ce-2a73-478f-abd3-7cce8763d9ab-kube-api-access-vsr7l\") pod \"847f16ce-2a73-478f-abd3-7cce8763d9ab\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") "
Jul 28 01:36:52.709191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709095    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f16ce-2a73-478f-abd3-7cce8763d9ab-tls-certs\") pod \"847f16ce-2a73-478f-abd3-7cce8763d9ab\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") "
Jul 28 01:36:52.709191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709187    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-home\") pod \"847f16ce-2a73-478f-abd3-7cce8763d9ab\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") "
Jul 28 01:36:52.709433 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709261    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-tmp-dir\") pod \"847f16ce-2a73-478f-abd3-7cce8763d9ab\" (UID: \"847f16ce-2a73-478f-abd3-7cce8763d9ab\") "
Jul 28 01:36:52.709433 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709363    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-model-cache" (OuterVolumeSpecName: "model-cache") pod "847f16ce-2a73-478f-abd3-7cce8763d9ab" (UID: "847f16ce-2a73-478f-abd3-7cce8763d9ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:36:52.709531 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709491    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "847f16ce-2a73-478f-abd3-7cce8763d9ab" (UID: "847f16ce-2a73-478f-abd3-7cce8763d9ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:36:52.709531 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709504    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-home" (OuterVolumeSpecName: "home") pod "847f16ce-2a73-478f-abd3-7cce8763d9ab" (UID: "847f16ce-2a73-478f-abd3-7cce8763d9ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:36:52.709615 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709598    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:52.709652 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709618    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:52.709652 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.709631    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:52.711126 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.711106    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/847f16ce-2a73-478f-abd3-7cce8763d9ab-kube-api-access-vsr7l" (OuterVolumeSpecName: "kube-api-access-vsr7l") pod "847f16ce-2a73-478f-abd3-7cce8763d9ab" (UID: "847f16ce-2a73-478f-abd3-7cce8763d9ab"). InnerVolumeSpecName "kube-api-access-vsr7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:52.711195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.711126    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/847f16ce-2a73-478f-abd3-7cce8763d9ab-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "847f16ce-2a73-478f-abd3-7cce8763d9ab" (UID: "847f16ce-2a73-478f-abd3-7cce8763d9ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:36:52.711195 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.711182    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-dshm" (OuterVolumeSpecName: "dshm") pod "847f16ce-2a73-478f-abd3-7cce8763d9ab" (UID: "847f16ce-2a73-478f-abd3-7cce8763d9ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:36:52.810359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.810321    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/847f16ce-2a73-478f-abd3-7cce8763d9ab-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:52.810359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.810357    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsr7l\" (UniqueName: \"kubernetes.io/projected/847f16ce-2a73-478f-abd3-7cce8763d9ab-kube-api-access-vsr7l\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:52.810698 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:52.810372    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/847f16ce-2a73-478f-abd3-7cce8763d9ab-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:53.288776 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.288745    2576 generic.go:356] "Generic (PLEG): container finished" podID="847f16ce-2a73-478f-abd3-7cce8763d9ab" containerID="c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8" exitCode=0
Jul 28 01:36:53.288930 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.288790    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"
Jul 28 01:36:53.288930 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.288827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk" event={"ID":"847f16ce-2a73-478f-abd3-7cce8763d9ab","Type":"ContainerDied","Data":"c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8"}
Jul 28 01:36:53.288930 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.288870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk" event={"ID":"847f16ce-2a73-478f-abd3-7cce8763d9ab","Type":"ContainerDied","Data":"0994ec091588d949c49831aad9ad9d5aeace0536b49f9945b5426bf97aa52e4a"}
Jul 28 01:36:53.288930 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.288893    2576 scope.go:117] "RemoveContainer" containerID="c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8"
Jul 28 01:36:53.308259 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.308240    2576 scope.go:117] "RemoveContainer" containerID="c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8"
Jul 28 01:36:53.308547 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:36:53.308528    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8\": container with ID starting with c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8 not found: ID does not exist" containerID="c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8"
Jul 28 01:36:53.308602 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.308558    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8"} err="failed to get container status \"c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8\": rpc error: code = NotFound desc = could not find container \"c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8\": container with ID starting with c2b1e835901f1e7f0103327fcf6bf788469cc67eb91a17aa7042c9494b24aed8 not found: ID does not exist"
Jul 28 01:36:53.316264 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.316237    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"]
Jul 28 01:36:53.320752 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.320715    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-3131e3/rate-test-model-3131e3-kserve-586958c8cf-zjspk"]
Jul 28 01:36:53.430293 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.430270    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:53.449825 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.449799    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7478-139833219896496_43d8c6cd-e28d-4426-86c4-62d4faea3030/test-curl-7478-139833219896496/0.log"
Jul 28 01:36:53.473300 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.473275    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:53.476406 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.476384    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:53.516046 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.516020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4t95q\" (UniqueName: \"kubernetes.io/projected/43d8c6cd-e28d-4426-86c4-62d4faea3030-kube-api-access-4t95q\") pod \"43d8c6cd-e28d-4426-86c4-62d4faea3030\" (UID: \"43d8c6cd-e28d-4426-86c4-62d4faea3030\") "
Jul 28 01:36:53.518034 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.518011    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d8c6cd-e28d-4426-86c4-62d4faea3030-kube-api-access-4t95q" (OuterVolumeSpecName: "kube-api-access-4t95q") pod "43d8c6cd-e28d-4426-86c4-62d4faea3030" (UID: "43d8c6cd-e28d-4426-86c4-62d4faea3030"). InnerVolumeSpecName "kube-api-access-4t95q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:53.591160 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591134    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:53.591527 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591514    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="847f16ce-2a73-478f-abd3-7cce8763d9ab" containerName="main"
Jul 28 01:36:53.591573 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591529    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="847f16ce-2a73-478f-abd3-7cce8763d9ab" containerName="main"
Jul 28 01:36:53.591573 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591545    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43d8c6cd-e28d-4426-86c4-62d4faea3030" containerName="test-curl-7478-139833219896496"
Jul 28 01:36:53.591573 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591551    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43d8c6cd-e28d-4426-86c4-62d4faea3030" containerName="test-curl-7478-139833219896496"
Jul 28 01:36:53.591679 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591616    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43d8c6cd-e28d-4426-86c4-62d4faea3030" containerName="test-curl-7478-139833219896496"
Jul 28 01:36:53.591679 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.591623    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="847f16ce-2a73-478f-abd3-7cce8763d9ab" containerName="main"
Jul 28 01:36:53.594552 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.594538    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:53.599230 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.599194    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:53.616962 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.616938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ntvv\" (UniqueName: \"kubernetes.io/projected/5a8c2f4c-3dc5-4390-99f5-5c783cca0499-kube-api-access-7ntvv\") pod \"test-curl-7478-139833219896496\" (UID: \"5a8c2f4c-3dc5-4390-99f5-5c783cca0499\") " pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:53.617064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.616999    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t95q\" (UniqueName: \"kubernetes.io/projected/43d8c6cd-e28d-4426-86c4-62d4faea3030-kube-api-access-4t95q\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:53.717740 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.717703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ntvv\" (UniqueName: \"kubernetes.io/projected/5a8c2f4c-3dc5-4390-99f5-5c783cca0499-kube-api-access-7ntvv\") pod \"test-curl-7478-139833219896496\" (UID: \"5a8c2f4c-3dc5-4390-99f5-5c783cca0499\") " pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:53.725878 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.725845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ntvv\" (UniqueName: \"kubernetes.io/projected/5a8c2f4c-3dc5-4390-99f5-5c783cca0499-kube-api-access-7ntvv\") pod \"test-curl-7478-139833219896496\" (UID: \"5a8c2f4c-3dc5-4390-99f5-5c783cca0499\") " pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:53.750556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.750528    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43d8c6cd-e28d-4426-86c4-62d4faea3030" path="/var/lib/kubelet/pods/43d8c6cd-e28d-4426-86c4-62d4faea3030/volumes"
Jul 28 01:36:53.751008 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.750988    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="847f16ce-2a73-478f-abd3-7cce8763d9ab" path="/var/lib/kubelet/pods/847f16ce-2a73-478f-abd3-7cce8763d9ab/volumes"
Jul 28 01:36:53.904078 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:53.903995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:54.029629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:54.029608    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:54.031726 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:36:54.031692    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8c2f4c_3dc5_4390_99f5_5c783cca0499.slice/crio-5c71addd7badec4b8a77f8b6fd608d5ca029da72cb790bf1e6f90b5f28aaa284 WatchSource:0}: Error finding container 5c71addd7badec4b8a77f8b6fd608d5ca029da72cb790bf1e6f90b5f28aaa284: Status 404 returned error can't find the container with id 5c71addd7badec4b8a77f8b6fd608d5ca029da72cb790bf1e6f90b5f28aaa284
Jul 28 01:36:54.296190 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:54.296114    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:54.296190 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:54.296122    2576 scope.go:117] "RemoveContainer" containerID="91a40c9614c82fa2e6593f969bd29f5d86c7a1308fef81efdb01b47ee0bc79c4"
Jul 28 01:36:54.297199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:54.297178    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219896496" event={"ID":"5a8c2f4c-3dc5-4390-99f5-5c783cca0499","Type":"ContainerStarted","Data":"5c71addd7badec4b8a77f8b6fd608d5ca029da72cb790bf1e6f90b5f28aaa284"}
Jul 28 01:36:55.303790 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:55.303747    2576 generic.go:356] "Generic (PLEG): container finished" podID="5a8c2f4c-3dc5-4390-99f5-5c783cca0499" containerID="9ba3aebfc6047e29eb9aca65949d011f3b781702b53e75d489cdc11d93fd5aab" exitCode=0
Jul 28 01:36:55.304184 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:55.303813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7478-139833219896496" event={"ID":"5a8c2f4c-3dc5-4390-99f5-5c783cca0499","Type":"ContainerDied","Data":"9ba3aebfc6047e29eb9aca65949d011f3b781702b53e75d489cdc11d93fd5aab"}
Jul 28 01:36:56.439378 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.439355    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:56.458205 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.458169    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7478-139833219896496_5a8c2f4c-3dc5-4390-99f5-5c783cca0499/test-curl-7478-139833219896496/0.log"
Jul 28 01:36:56.477753 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.477725    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:56.481151 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.481115    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7478-139833219896496"]
Jul 28 01:36:56.540042 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.540016    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ntvv\" (UniqueName: \"kubernetes.io/projected/5a8c2f4c-3dc5-4390-99f5-5c783cca0499-kube-api-access-7ntvv\") pod \"5a8c2f4c-3dc5-4390-99f5-5c783cca0499\" (UID: \"5a8c2f4c-3dc5-4390-99f5-5c783cca0499\") "
Jul 28 01:36:56.542031 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.542006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a8c2f4c-3dc5-4390-99f5-5c783cca0499-kube-api-access-7ntvv" (OuterVolumeSpecName: "kube-api-access-7ntvv") pod "5a8c2f4c-3dc5-4390-99f5-5c783cca0499" (UID: "5a8c2f4c-3dc5-4390-99f5-5c783cca0499"). InnerVolumeSpecName "kube-api-access-7ntvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:36:56.640843 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.640776    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ntvv\" (UniqueName: \"kubernetes.io/projected/5a8c2f4c-3dc5-4390-99f5-5c783cca0499-kube-api-access-7ntvv\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:36:56.953218 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:56.953091    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:57.248376 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:57.248297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:57.313950 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:57.313920    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:57.314106 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:57.313961    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c71addd7badec4b8a77f8b6fd608d5ca029da72cb790bf1e6f90b5f28aaa284"
Jul 28 01:36:57.323959 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:36:57.323933    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7478-139833219896496\" is forbidden: User \"system:node:ip-10-0-140-154.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-154.ec2.internal' and this object" podUID="5a8c2f4c-3dc5-4390-99f5-5c783cca0499" pod="opendatahub/test-curl-7478-139833219896496"
Jul 28 01:36:57.441660 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:57.441614    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:57.751808 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:57.751777    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8c2f4c-3dc5-4390-99f5-5c783cca0499" path="/var/lib/kubelet/pods/5a8c2f4c-3dc5-4390-99f5-5c783cca0499/volumes"
Jul 28 01:36:59.822011 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:59.821969    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:59.920515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:36:59.920488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:00.043283 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.043248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:00.373683 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.373641    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:00.497476 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.497444    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:00.600373 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.600337    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:00.607735 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.607705    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"]
Jul 28 01:37:00.608084 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.608071    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8c2f4c-3dc5-4390-99f5-5c783cca0499" containerName="test-curl-7478-139833219896496"
Jul 28 01:37:00.608129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.608087    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8c2f4c-3dc5-4390-99f5-5c783cca0499" containerName="test-curl-7478-139833219896496"
Jul 28 01:37:00.608172 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.608156    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8c2f4c-3dc5-4390-99f5-5c783cca0499" containerName="test-curl-7478-139833219896496"
Jul 28 01:37:00.611331 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.611314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.614065 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.614047    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c0266d\""
Jul 28 01:37:00.614186 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.614135    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c0266d-dockercfg-rnb4g\""
Jul 28 01:37:00.620959 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.620934    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"]
Jul 28 01:37:00.622888 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.622862    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"]
Jul 28 01:37:00.626515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.626464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.635449 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.635429    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"]
Jul 28 01:37:00.675756 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.675725    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9s4p\" (UniqueName: \"kubernetes.io/projected/fc58796a-b674-47dc-bf92-e4bf933fbd30-kube-api-access-h9s4p\") pod \"payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.675896 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.675773    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4109b-f81a-4be8-af77-606da8599936-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.675896 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.675799    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfgpm\" (UniqueName: \"kubernetes.io/projected/f7e4109b-f81a-4be8-af77-606da8599936-kube-api-access-cfgpm\") pod \"payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.675896 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.675847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc58796a-b674-47dc-bf92-e4bf933fbd30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.777187 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.777154    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4109b-f81a-4be8-af77-606da8599936-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.777391 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.777192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfgpm\" (UniqueName: \"kubernetes.io/projected/f7e4109b-f81a-4be8-af77-606da8599936-kube-api-access-cfgpm\") pod \"payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.777391 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.777258    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc58796a-b674-47dc-bf92-e4bf933fbd30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.777391 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.777301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9s4p\" (UniqueName: \"kubernetes.io/projected/fc58796a-b674-47dc-bf92-e4bf933fbd30-kube-api-access-h9s4p\") pod \"payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.777919 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.777894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4109b-f81a-4be8-af77-606da8599936-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.777996 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.777957    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc58796a-b674-47dc-bf92-e4bf933fbd30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.785375 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.785352    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9s4p\" (UniqueName: \"kubernetes.io/projected/fc58796a-b674-47dc-bf92-e4bf933fbd30-kube-api-access-h9s4p\") pod \"payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:00.785614 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.785597    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfgpm\" (UniqueName: \"kubernetes.io/projected/f7e4109b-f81a-4be8-af77-606da8599936-kube-api-access-cfgpm\") pod \"payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.923003 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.922943    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:00.938025 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:00.938002    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:01.063085 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.063051    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"]
Jul 28 01:37:01.064121 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:37:01.064098    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e4109b_f81a_4be8_af77_606da8599936.slice/crio-08bf4d8f300caa1ab91b0806984488024a9567594ae73444871071b84de3007c WatchSource:0}: Error finding container 08bf4d8f300caa1ab91b0806984488024a9567594ae73444871071b84de3007c: Status 404 returned error can't find the container with id 08bf4d8f300caa1ab91b0806984488024a9567594ae73444871071b84de3007c
Jul 28 01:37:01.300786 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.300761    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"]
Jul 28 01:37:01.302881 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:37:01.302852    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc58796a_b674_47dc_bf92_e4bf933fbd30.slice/crio-6acb0f15e3b3edcf7b58590ada5b872fba503404f4608b0840ba3f811406860d WatchSource:0}: Error finding container 6acb0f15e3b3edcf7b58590ada5b872fba503404f4608b0840ba3f811406860d: Status 404 returned error can't find the container with id 6acb0f15e3b3edcf7b58590ada5b872fba503404f4608b0840ba3f811406860d
Jul 28 01:37:01.335667 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.335640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs" event={"ID":"fc58796a-b674-47dc-bf92-e4bf933fbd30","Type":"ContainerStarted","Data":"6acb0f15e3b3edcf7b58590ada5b872fba503404f4608b0840ba3f811406860d"}
Jul 28 01:37:01.337234 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.337195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v" event={"ID":"f7e4109b-f81a-4be8-af77-606da8599936","Type":"ContainerStarted","Data":"45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096"}
Jul 28 01:37:01.337341 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.337267    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:01.337341 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.337289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v" event={"ID":"f7e4109b-f81a-4be8-af77-606da8599936","Type":"ContainerStarted","Data":"08bf4d8f300caa1ab91b0806984488024a9567594ae73444871071b84de3007c"}
Jul 28 01:37:01.356462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:01.356408    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v" podStartSLOduration=1.3563867969999999 podStartE2EDuration="1.356386797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:01.354962663 +0000 UTC m=+3026.104101386" watchObservedRunningTime="2026-07-28 01:37:01.356386797 +0000 UTC m=+3026.105525523"
Jul 28 01:37:02.344075 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:02.344040    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs" event={"ID":"fc58796a-b674-47dc-bf92-e4bf933fbd30","Type":"ContainerStarted","Data":"90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297"}
Jul 28 01:37:03.349391 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:03.349353    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:04.302712 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:04.302677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:12.346565 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:12.346532    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:37:12.364172 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:12.364120    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs" podStartSLOduration=12.364105921 podStartE2EDuration="12.364105921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:02.360536453 +0000 UTC m=+3027.109675179" watchObservedRunningTime="2026-07-28 01:37:12.364105921 +0000 UTC m=+3037.113244646"
Jul 28 01:37:12.599382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:12.599301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:14.357815 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:14.357785    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:37:15.194424 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:15.194394    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:15.522973 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:15.522899    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:15.643183 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:15.643146    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:18.199641 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:18.199607    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:18.286464 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:18.286421    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:18.387081 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:18.387044    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:18.697464 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:18.697430    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:18.951049 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:18.950960    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:19.054664 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.054632    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:19.179486 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.179455    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"]
Jul 28 01:37:19.182881 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.182864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.185871 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.185849    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1ce59a\""
Jul 28 01:37:19.185979 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.185935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1ce59a-dockercfg-x8ckg\""
Jul 28 01:37:19.205393 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.205341    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"]
Jul 28 01:37:19.273286 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.273258    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"]
Jul 28 01:37:19.276794 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.276774    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.291071 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.291045    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"]
Jul 28 01:37:19.336670 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.336650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.336768 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.336692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8wb\" (UniqueName: \"kubernetes.io/projected/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-kube-api-access-8w8wb\") pod \"payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.437351 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.437328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2928a61-07c4-4df6-803d-2936f8e2fb87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.437465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.437386    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.437465 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.437431    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8wb\" (UniqueName: \"kubernetes.io/projected/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-kube-api-access-8w8wb\") pod \"payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.437543 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.437463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd24v\" (UniqueName: \"kubernetes.io/projected/a2928a61-07c4-4df6-803d-2936f8e2fb87-kube-api-access-zd24v\") pod \"payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.438018 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.438002    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.449980 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.449955    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8wb\" (UniqueName: \"kubernetes.io/projected/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-kube-api-access-8w8wb\") pod \"payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.493931 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.493864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:19.538367 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.538334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2928a61-07c4-4df6-803d-2936f8e2fb87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.538543 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.538415    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd24v\" (UniqueName: \"kubernetes.io/projected/a2928a61-07c4-4df6-803d-2936f8e2fb87-kube-api-access-zd24v\") pod \"payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.539163 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.539140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2928a61-07c4-4df6-803d-2936f8e2fb87-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.548824 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.548792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd24v\" (UniqueName: \"kubernetes.io/projected/a2928a61-07c4-4df6-803d-2936f8e2fb87-kube-api-access-zd24v\") pod \"payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.586550 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.586521    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:19.633970 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.633937    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"]
Jul 28 01:37:19.635786 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:37:19.635760    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff76ee4_28d8_4ea9_941e_2e748dfb3485.slice/crio-1929a91af7665a12678d4725e7e56738e8d8afd4c4cc028cb08a00a622e3f202 WatchSource:0}: Error finding container 1929a91af7665a12678d4725e7e56738e8d8afd4c4cc028cb08a00a622e3f202: Status 404 returned error can't find the container with id 1929a91af7665a12678d4725e7e56738e8d8afd4c4cc028cb08a00a622e3f202
Jul 28 01:37:19.945784 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:19.945760    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"]
Jul 28 01:37:19.947000 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:37:19.946976    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2928a61_07c4_4df6_803d_2936f8e2fb87.slice/crio-e38fad9b8c06d2a8bdf7de7fdbdabf436597e6eeb80828dc3d6c43b7a90cafea WatchSource:0}: Error finding container e38fad9b8c06d2a8bdf7de7fdbdabf436597e6eeb80828dc3d6c43b7a90cafea: Status 404 returned error can't find the container with id e38fad9b8c06d2a8bdf7de7fdbdabf436597e6eeb80828dc3d6c43b7a90cafea
Jul 28 01:37:20.434853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.434812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r" event={"ID":"a2928a61-07c4-4df6-803d-2936f8e2fb87","Type":"ContainerStarted","Data":"5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94"}
Jul 28 01:37:20.435325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.434878    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:20.435325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.434899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r" event={"ID":"a2928a61-07c4-4df6-803d-2936f8e2fb87","Type":"ContainerStarted","Data":"e38fad9b8c06d2a8bdf7de7fdbdabf436597e6eeb80828dc3d6c43b7a90cafea"}
Jul 28 01:37:20.436330 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.436305    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd" event={"ID":"7ff76ee4-28d8-4ea9-941e-2e748dfb3485","Type":"ContainerStarted","Data":"f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5"}
Jul 28 01:37:20.436434 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.436346    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd" event={"ID":"7ff76ee4-28d8-4ea9-941e-2e748dfb3485","Type":"ContainerStarted","Data":"1929a91af7665a12678d4725e7e56738e8d8afd4c4cc028cb08a00a622e3f202"}
Jul 28 01:37:20.436474 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.436437    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:20.451723 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.451684    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r" podStartSLOduration=1.451669991 podStartE2EDuration="1.451669991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:20.44984919 +0000 UTC m=+3045.198987915" watchObservedRunningTime="2026-07-28 01:37:20.451669991 +0000 UTC m=+3045.200808714"
Jul 28 01:37:20.464506 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:20.464459    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd" podStartSLOduration=1.464441724 podStartE2EDuration="1.464441724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:20.464364126 +0000 UTC m=+3045.213502854" watchObservedRunningTime="2026-07-28 01:37:20.464441724 +0000 UTC m=+3045.213580448"
Jul 28 01:37:22.558917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:22.558884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:31.443660 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:31.443404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:37:31.444086 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:31.443762    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:37:39.661891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:39.661859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vtbwh_7041fa74-1187-4ba0-aa28-1405298481c6/payload-processing/0.log"
Jul 28 01:37:39.785109 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:39.785068    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r_a2928a61-07c4-4df6-803d-2936f8e2fb87/payload-processing/0.log"
Jul 28 01:37:52.632036 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.632001    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"]
Jul 28 01:37:52.636069 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.636011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.638978 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.638950    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c0266d\"/\"openshift-service-ca.crt\""
Jul 28 01:37:52.640303 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.640282    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c0266d\"/\"ipp-route-c0266d-kserve-self-signed-certs\""
Jul 28 01:37:52.640406 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.640378    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c0266d\"/\"default-dockercfg-47nz5\""
Jul 28 01:37:52.640474 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.640428    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c0266d\"/\"kube-root-ca.crt\""
Jul 28 01:37:52.650107 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.650083    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"]
Jul 28 01:37:52.683500 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.683476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:52.726960 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.726937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf57z\" (UniqueName: \"kubernetes.io/projected/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-kube-api-access-sf57z\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.727069 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.726982    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tls-certs\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.727069 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.727002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-home\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.727191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.727077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-model-cache\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.727191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.727110    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tmp-dir\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.727191 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.727142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-dshm\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828315 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828290    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-model-cache\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828437 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tmp-dir\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828437 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-dshm\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828731 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf57z\" (UniqueName: \"kubernetes.io/projected/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-kube-api-access-sf57z\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828817 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tmp-dir\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828817 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828807    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-model-cache\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828928 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tls-certs\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.828928 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.828855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-home\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.829133 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.829115    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-home\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.830625 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.830600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-dshm\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.830944 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.830928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tls-certs\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.836733 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.836715    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf57z\" (UniqueName: \"kubernetes.io/projected/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-kube-api-access-sf57z\") pod \"ipp-route-c0266d-kserve-5f7fcbb547-2wp9s\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") " pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:52.947082 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:52.947004    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:37:53.085137 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:53.085105    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"]
Jul 28 01:37:53.088181 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:37:53.088149    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6ebf3ed_6bd4_42eb_84dc_ce98d178f137.slice/crio-41e35c15cd0b770596111250e580b4b250ba932f92ec7c274fd53576fe3a4757 WatchSource:0}: Error finding container 41e35c15cd0b770596111250e580b4b250ba932f92ec7c274fd53576fe3a4757: Status 404 returned error can't find the container with id 41e35c15cd0b770596111250e580b4b250ba932f92ec7c274fd53576fe3a4757
Jul 28 01:37:53.607920 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:53.607883    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s" event={"ID":"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137","Type":"ContainerStarted","Data":"9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a"}
Jul 28 01:37:53.608099 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:53.607926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s" event={"ID":"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137","Type":"ContainerStarted","Data":"41e35c15cd0b770596111250e580b4b250ba932f92ec7c274fd53576fe3a4757"}
Jul 28 01:37:53.626468 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:37:53.626413    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s" podStartSLOduration=1.626398233 podStartE2EDuration="1.626398233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:53.624921288 +0000 UTC m=+3078.374060012" watchObservedRunningTime="2026-07-28 01:37:53.626398233 +0000 UTC m=+3078.375537015"
Jul 28 01:38:02.947391 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:02.947358    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:38:02.947799 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:02.947406    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:38:02.960351 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:02.960326    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:38:03.670001 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:03.669971    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:38:37.094747 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:37.094713    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs_fc58796a-b674-47dc-bf92-e4bf933fbd30/payload-processing/0.log"
Jul 28 01:38:37.224659 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:37.224633    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vtbwh_7041fa74-1187-4ba0-aa28-1405298481c6/payload-processing/0.log"
Jul 28 01:38:37.347164 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:37.347095    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r_a2928a61-07c4-4df6-803d-2936f8e2fb87/payload-processing/0.log"
Jul 28 01:38:37.633549 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:37.633472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:37.959462 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:37.959368    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:38.262847 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:38.262768    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:40.576239 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:40.576200    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:40.669638 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:40.669602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:41.638553 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:41.638523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:41.961059 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:41.960977    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:42.088491 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.088456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:42.180547 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.180513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:42.186091 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.186063    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"]
Jul 28 01:38:42.189917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.189900    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.192530 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.192510    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5ecc77-dockercfg-7s2tf\""
Jul 28 01:38:42.192627 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.192536    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5ecc77\""
Jul 28 01:38:42.199559 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.199539    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"]
Jul 28 01:38:42.229627 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.229575    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"]
Jul 28 01:38:42.232837 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.232822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.242720 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.242699    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"]
Jul 28 01:38:42.255159 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.255136    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b753e0d-b75f-4136-b5f2-cb98cefa579a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.255282 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.255194    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2q5j\" (UniqueName: \"kubernetes.io/projected/9b753e0d-b75f-4136-b5f2-cb98cefa579a-kube-api-access-v2q5j\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.355762 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.355729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2q5j\" (UniqueName: \"kubernetes.io/projected/9b753e0d-b75f-4136-b5f2-cb98cefa579a-kube-api-access-v2q5j\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.355902 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.355872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400488e7-c439-45e6-aa72-f4e28bcf3cdb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.355972 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.355910    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b753e0d-b75f-4136-b5f2-cb98cefa579a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.356053 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.355976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/400488e7-c439-45e6-aa72-f4e28bcf3cdb-kube-api-access-z8z4v\") pod \"payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.356544 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.356527    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b753e0d-b75f-4136-b5f2-cb98cefa579a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.364237 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.364209    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2q5j\" (UniqueName: \"kubernetes.io/projected/9b753e0d-b75f-4136-b5f2-cb98cefa579a-kube-api-access-v2q5j\") pod \"payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.457254 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.457210    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400488e7-c439-45e6-aa72-f4e28bcf3cdb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.457389 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.457274    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/400488e7-c439-45e6-aa72-f4e28bcf3cdb-kube-api-access-z8z4v\") pod \"payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.457852 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.457832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400488e7-c439-45e6-aa72-f4e28bcf3cdb-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.479521 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.479498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/400488e7-c439-45e6-aa72-f4e28bcf3cdb-kube-api-access-z8z4v\") pod \"payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.502193 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.502140    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.542185 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.542158    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:42.646968 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.646888    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"]
Jul 28 01:38:42.650717 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:38:42.650686    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b753e0d_b75f_4136_b5f2_cb98cefa579a.slice/crio-748d7205e8e1ccfd023085bb03a1072e19608f61b9f9cc58ff335a085dca50e8 WatchSource:0}: Error finding container 748d7205e8e1ccfd023085bb03a1072e19608f61b9f9cc58ff335a085dca50e8: Status 404 returned error can't find the container with id 748d7205e8e1ccfd023085bb03a1072e19608f61b9f9cc58ff335a085dca50e8
Jul 28 01:38:42.863004 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.862969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478" event={"ID":"9b753e0d-b75f-4136-b5f2-cb98cefa579a","Type":"ContainerStarted","Data":"3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678"}
Jul 28 01:38:42.863180 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.863033    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478" event={"ID":"9b753e0d-b75f-4136-b5f2-cb98cefa579a","Type":"ContainerStarted","Data":"748d7205e8e1ccfd023085bb03a1072e19608f61b9f9cc58ff335a085dca50e8"}
Jul 28 01:38:42.863180 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.863056    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:42.880587 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.880541    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478" podStartSLOduration=0.880525193 podStartE2EDuration="880.525193ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:42.878580194 +0000 UTC m=+3127.627718920" watchObservedRunningTime="2026-07-28 01:38:42.880525193 +0000 UTC m=+3127.629663926"
Jul 28 01:38:42.899972 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:42.899951    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"]
Jul 28 01:38:42.901660 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:38:42.901637    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400488e7_c439_45e6_aa72_f4e28bcf3cdb.slice/crio-3756f9b7de73601c1e93fb25ed1b5d0ef958982a9c0b7e197e86e52caad6ce78 WatchSource:0}: Error finding container 3756f9b7de73601c1e93fb25ed1b5d0ef958982a9c0b7e197e86e52caad6ce78: Status 404 returned error can't find the container with id 3756f9b7de73601c1e93fb25ed1b5d0ef958982a9c0b7e197e86e52caad6ce78
Jul 28 01:38:43.870238 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:43.870190    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt" event={"ID":"400488e7-c439-45e6-aa72-f4e28bcf3cdb","Type":"ContainerStarted","Data":"683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232"}
Jul 28 01:38:43.870656 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:43.870272    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:38:43.870656 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:43.870300    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt" event={"ID":"400488e7-c439-45e6-aa72-f4e28bcf3cdb","Type":"ContainerStarted","Data":"3756f9b7de73601c1e93fb25ed1b5d0ef958982a9c0b7e197e86e52caad6ce78"}
Jul 28 01:38:43.888120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:43.888063    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt" podStartSLOduration=1.8880430769999998 podStartE2EDuration="1.888043077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:43.885695851 +0000 UTC m=+3128.634834576" watchObservedRunningTime="2026-07-28 01:38:43.888043077 +0000 UTC m=+3128.637181801"
Jul 28 01:38:45.055382 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:45.055340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:53.872856 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:53.872821    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:38:54.877599 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:38:54.877569    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:39:06.259325 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:06.259289    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:06.367702 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:06.367672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:07.366199 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.366157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:07.390845 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.390809    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"]
Jul 28 01:39:07.391096 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.391063    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt" podUID="400488e7-c439-45e6-aa72-f4e28bcf3cdb" containerName="payload-processing" containerID="cri-o://683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232" gracePeriod=30
Jul 28 01:39:07.396430 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.396406    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"]
Jul 28 01:39:07.396812 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.396779    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478" podUID="9b753e0d-b75f-4136-b5f2-cb98cefa579a" containerName="payload-pre-processing" containerID="cri-o://3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678" gracePeriod=30
Jul 28 01:39:07.430548 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.430521    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:07.492956 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.492928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:07.663229 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.663199    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:39:07.679665 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.679575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400488e7-c439-45e6-aa72-f4e28bcf3cdb-plugins-config-volume\") pod \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") "
Jul 28 01:39:07.679793 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.679719    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/400488e7-c439-45e6-aa72-f4e28bcf3cdb-kube-api-access-z8z4v\") pod \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\" (UID: \"400488e7-c439-45e6-aa72-f4e28bcf3cdb\") "
Jul 28 01:39:07.681284 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.681190    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/400488e7-c439-45e6-aa72-f4e28bcf3cdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "400488e7-c439-45e6-aa72-f4e28bcf3cdb" (UID: "400488e7-c439-45e6-aa72-f4e28bcf3cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:07.683131 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.683098    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/400488e7-c439-45e6-aa72-f4e28bcf3cdb-kube-api-access-z8z4v" (OuterVolumeSpecName: "kube-api-access-z8z4v") pod "400488e7-c439-45e6-aa72-f4e28bcf3cdb" (UID: "400488e7-c439-45e6-aa72-f4e28bcf3cdb"). InnerVolumeSpecName "kube-api-access-z8z4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:07.689668 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.689646    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:39:07.780543 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.780517    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b753e0d-b75f-4136-b5f2-cb98cefa579a-plugins-config-volume\") pod \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") "
Jul 28 01:39:07.780671 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.780563    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2q5j\" (UniqueName: \"kubernetes.io/projected/9b753e0d-b75f-4136-b5f2-cb98cefa579a-kube-api-access-v2q5j\") pod \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\" (UID: \"9b753e0d-b75f-4136-b5f2-cb98cefa579a\") "
Jul 28 01:39:07.780817 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.780800    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8z4v\" (UniqueName: \"kubernetes.io/projected/400488e7-c439-45e6-aa72-f4e28bcf3cdb-kube-api-access-z8z4v\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:07.780872 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.780822    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/400488e7-c439-45e6-aa72-f4e28bcf3cdb-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:07.780872 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.780846    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b753e0d-b75f-4136-b5f2-cb98cefa579a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9b753e0d-b75f-4136-b5f2-cb98cefa579a" (UID: "9b753e0d-b75f-4136-b5f2-cb98cefa579a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:07.782630 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.782605    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b753e0d-b75f-4136-b5f2-cb98cefa579a-kube-api-access-v2q5j" (OuterVolumeSpecName: "kube-api-access-v2q5j") pod "9b753e0d-b75f-4136-b5f2-cb98cefa579a" (UID: "9b753e0d-b75f-4136-b5f2-cb98cefa579a"). InnerVolumeSpecName "kube-api-access-v2q5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:07.881856 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.881833    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b753e0d-b75f-4136-b5f2-cb98cefa579a-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:07.881856 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:07.881854    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2q5j\" (UniqueName: \"kubernetes.io/projected/9b753e0d-b75f-4136-b5f2-cb98cefa579a-kube-api-access-v2q5j\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:08.001792 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.001721    2576 generic.go:356] "Generic (PLEG): container finished" podID="9b753e0d-b75f-4136-b5f2-cb98cefa579a" containerID="3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678" exitCode=0
Jul 28 01:39:08.001792 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.001773    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"
Jul 28 01:39:08.001953 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.001801    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478" event={"ID":"9b753e0d-b75f-4136-b5f2-cb98cefa579a","Type":"ContainerDied","Data":"3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678"}
Jul 28 01:39:08.001953 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.001862    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478" event={"ID":"9b753e0d-b75f-4136-b5f2-cb98cefa579a","Type":"ContainerDied","Data":"748d7205e8e1ccfd023085bb03a1072e19608f61b9f9cc58ff335a085dca50e8"}
Jul 28 01:39:08.001953 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.001898    2576 scope.go:117] "RemoveContainer" containerID="3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678"
Jul 28 01:39:08.003153 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.003129    2576 generic.go:356] "Generic (PLEG): container finished" podID="400488e7-c439-45e6-aa72-f4e28bcf3cdb" containerID="683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232" exitCode=0
Jul 28 01:39:08.003257 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.003202    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"
Jul 28 01:39:08.003328 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.003248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt" event={"ID":"400488e7-c439-45e6-aa72-f4e28bcf3cdb","Type":"ContainerDied","Data":"683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232"}
Jul 28 01:39:08.003328 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.003299    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt" event={"ID":"400488e7-c439-45e6-aa72-f4e28bcf3cdb","Type":"ContainerDied","Data":"3756f9b7de73601c1e93fb25ed1b5d0ef958982a9c0b7e197e86e52caad6ce78"}
Jul 28 01:39:08.013369 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.013310    2576 scope.go:117] "RemoveContainer" containerID="3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678"
Jul 28 01:39:08.013710 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:39:08.013691    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678\": container with ID starting with 3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678 not found: ID does not exist" containerID="3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678"
Jul 28 01:39:08.013787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.013718    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678"} err="failed to get container status \"3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678\": rpc error: code = NotFound desc = could not find container \"3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678\": container with ID starting with 3b5a8ecf7304970a11e49a251495592d470cafecb8917926b1bc5f420bfe2678 not found: ID does not exist"
Jul 28 01:39:08.013787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.013737    2576 scope.go:117] "RemoveContainer" containerID="683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232"
Jul 28 01:39:08.021362 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.021346    2576 scope.go:117] "RemoveContainer" containerID="683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232"
Jul 28 01:39:08.021616 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:39:08.021587    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232\": container with ID starting with 683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232 not found: ID does not exist" containerID="683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232"
Jul 28 01:39:08.021669 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.021616    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232"} err="failed to get container status \"683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232\": rpc error: code = NotFound desc = could not find container \"683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232\": container with ID starting with 683762f1825b1e4735f6575387a48ee601416abebf9490fc85764e705f166232 not found: ID does not exist"
Jul 28 01:39:08.035776 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.035753    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"]
Jul 28 01:39:08.049540 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.049518    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-5ecc77-84d5dd85cb-dfnrt"]
Jul 28 01:39:08.075765 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.075745    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"]
Jul 28 01:39:08.084878 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:08.084858    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5ecc77-7b475db44-r4478"]
Jul 28 01:39:09.752147 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:09.752109    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="400488e7-c439-45e6-aa72-f4e28bcf3cdb" path="/var/lib/kubelet/pods/400488e7-c439-45e6-aa72-f4e28bcf3cdb/volumes"
Jul 28 01:39:09.752567 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:09.752545    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b753e0d-b75f-4136-b5f2-cb98cefa579a" path="/var/lib/kubelet/pods/9b753e0d-b75f-4136-b5f2-cb98cefa579a/volumes"
Jul 28 01:39:11.120809 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:11.120775    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:11.504267 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:11.504164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:17.940133 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:17.940097    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:17.998657 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:17.998603    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:18.236715 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:18.236641    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:18.460700 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:18.460668    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:31.263019 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:31.262983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:39.762062 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.762025    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:39.790104 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.790074    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"]
Jul 28 01:39:39.790425 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.790389    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs" podUID="fc58796a-b674-47dc-bf92-e4bf933fbd30" containerName="payload-processing" containerID="cri-o://90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297" gracePeriod=30
Jul 28 01:39:39.797568 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.797541    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"]
Jul 28 01:39:39.797769 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.797722    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v" podUID="f7e4109b-f81a-4be8-af77-606da8599936" containerName="payload-pre-processing" containerID="cri-o://45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096" gracePeriod=30
Jul 28 01:39:39.831553 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.831533    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:39.898296 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:39.898267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:40.082391 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.082359    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:39:40.085137 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.085118    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:39:40.171944 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.171910    2576 generic.go:356] "Generic (PLEG): container finished" podID="fc58796a-b674-47dc-bf92-e4bf933fbd30" containerID="90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297" exitCode=0
Jul 28 01:39:40.172120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.171968    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"
Jul 28 01:39:40.172120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.172003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs" event={"ID":"fc58796a-b674-47dc-bf92-e4bf933fbd30","Type":"ContainerDied","Data":"90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297"}
Jul 28 01:39:40.172120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.172092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs" event={"ID":"fc58796a-b674-47dc-bf92-e4bf933fbd30","Type":"ContainerDied","Data":"6acb0f15e3b3edcf7b58590ada5b872fba503404f4608b0840ba3f811406860d"}
Jul 28 01:39:40.172120 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.172120    2576 scope.go:117] "RemoveContainer" containerID="90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297"
Jul 28 01:39:40.173494 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.173467    2576 generic.go:356] "Generic (PLEG): container finished" podID="f7e4109b-f81a-4be8-af77-606da8599936" containerID="45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096" exitCode=0
Jul 28 01:39:40.173602 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.173518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v" event={"ID":"f7e4109b-f81a-4be8-af77-606da8599936","Type":"ContainerDied","Data":"45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096"}
Jul 28 01:39:40.173602 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.173589    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v" event={"ID":"f7e4109b-f81a-4be8-af77-606da8599936","Type":"ContainerDied","Data":"08bf4d8f300caa1ab91b0806984488024a9567594ae73444871071b84de3007c"}
Jul 28 01:39:40.173705 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.173541    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"
Jul 28 01:39:40.181491 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.181472    2576 scope.go:117] "RemoveContainer" containerID="90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297"
Jul 28 01:39:40.181737 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:39:40.181722    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297\": container with ID starting with 90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297 not found: ID does not exist" containerID="90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297"
Jul 28 01:39:40.181785 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.181743    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297"} err="failed to get container status \"90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297\": rpc error: code = NotFound desc = could not find container \"90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297\": container with ID starting with 90381a4d34cab721ccebfcf24beb2fa2af79651eabc392fa5de7902364ce9297 not found: ID does not exist"
Jul 28 01:39:40.181785 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.181758    2576 scope.go:117] "RemoveContainer" containerID="45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096"
Jul 28 01:39:40.189094 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.189080    2576 scope.go:117] "RemoveContainer" containerID="45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096"
Jul 28 01:39:40.189346 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:39:40.189327    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096\": container with ID starting with 45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096 not found: ID does not exist" containerID="45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096"
Jul 28 01:39:40.189416 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.189354    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096"} err="failed to get container status \"45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096\": rpc error: code = NotFound desc = could not find container \"45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096\": container with ID starting with 45d689009f416a4e4d18c6efb30a9c3a779e2f52730b3fbf1e435e84ce208096 not found: ID does not exist"
Jul 28 01:39:40.270538 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.270512    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfgpm\" (UniqueName: \"kubernetes.io/projected/f7e4109b-f81a-4be8-af77-606da8599936-kube-api-access-cfgpm\") pod \"f7e4109b-f81a-4be8-af77-606da8599936\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") "
Jul 28 01:39:40.270645 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.270557    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc58796a-b674-47dc-bf92-e4bf933fbd30-plugins-config-volume\") pod \"fc58796a-b674-47dc-bf92-e4bf933fbd30\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") "
Jul 28 01:39:40.270645 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.270629    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4109b-f81a-4be8-af77-606da8599936-plugins-config-volume\") pod \"f7e4109b-f81a-4be8-af77-606da8599936\" (UID: \"f7e4109b-f81a-4be8-af77-606da8599936\") "
Jul 28 01:39:40.270756 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.270695    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9s4p\" (UniqueName: \"kubernetes.io/projected/fc58796a-b674-47dc-bf92-e4bf933fbd30-kube-api-access-h9s4p\") pod \"fc58796a-b674-47dc-bf92-e4bf933fbd30\" (UID: \"fc58796a-b674-47dc-bf92-e4bf933fbd30\") "
Jul 28 01:39:40.270962 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.270942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc58796a-b674-47dc-bf92-e4bf933fbd30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc58796a-b674-47dc-bf92-e4bf933fbd30" (UID: "fc58796a-b674-47dc-bf92-e4bf933fbd30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:40.271069 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.271035    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7e4109b-f81a-4be8-af77-606da8599936-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7e4109b-f81a-4be8-af77-606da8599936" (UID: "f7e4109b-f81a-4be8-af77-606da8599936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:39:40.272570 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.272549    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e4109b-f81a-4be8-af77-606da8599936-kube-api-access-cfgpm" (OuterVolumeSpecName: "kube-api-access-cfgpm") pod "f7e4109b-f81a-4be8-af77-606da8599936" (UID: "f7e4109b-f81a-4be8-af77-606da8599936"). InnerVolumeSpecName "kube-api-access-cfgpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:40.272630 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.272572    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc58796a-b674-47dc-bf92-e4bf933fbd30-kube-api-access-h9s4p" (OuterVolumeSpecName: "kube-api-access-h9s4p") pod "fc58796a-b674-47dc-bf92-e4bf933fbd30" (UID: "fc58796a-b674-47dc-bf92-e4bf933fbd30"). InnerVolumeSpecName "kube-api-access-h9s4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:39:40.371889 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.371866    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7e4109b-f81a-4be8-af77-606da8599936-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.371889 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.371886    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9s4p\" (UniqueName: \"kubernetes.io/projected/fc58796a-b674-47dc-bf92-e4bf933fbd30-kube-api-access-h9s4p\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.372041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.371896    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfgpm\" (UniqueName: \"kubernetes.io/projected/f7e4109b-f81a-4be8-af77-606da8599936-kube-api-access-cfgpm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.372041 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.371904    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc58796a-b674-47dc-bf92-e4bf933fbd30-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:39:40.496386 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.496359    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"]
Jul 28 01:39:40.500472 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.500449    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c0266d-54f775cd56-qhw9v"]
Jul 28 01:39:40.510820 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.510799    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"]
Jul 28 01:39:40.514702 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:40.514683    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c0266d-8bcc6d46d-x8gfs"]
Jul 28 01:39:41.752847 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:41.752815    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e4109b-f81a-4be8-af77-606da8599936" path="/var/lib/kubelet/pods/f7e4109b-f81a-4be8-af77-606da8599936/volumes"
Jul 28 01:39:41.753263 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:41.753180    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc58796a-b674-47dc-bf92-e4bf933fbd30" path="/var/lib/kubelet/pods/fc58796a-b674-47dc-bf92-e4bf933fbd30/volumes"
Jul 28 01:39:42.016985 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:42.016893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:42.414622 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:42.414594    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:47.395595 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:47.395557    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"]
Jul 28 01:39:47.494002 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:39:47.493909    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:02.415016 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.414960    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s" podUID="a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" containerName="main" containerID="cri-o://9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a" gracePeriod=45
Jul 28 01:40:02.669114 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.669051    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:40:02.763377 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763345    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf57z\" (UniqueName: \"kubernetes.io/projected/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-kube-api-access-sf57z\") pod \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") "
Jul 28 01:40:02.763554 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763405    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tmp-dir\") pod \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") "
Jul 28 01:40:02.763554 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763531    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-model-cache\") pod \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") "
Jul 28 01:40:02.763685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763562    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-dshm\") pod \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") "
Jul 28 01:40:02.763685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763630    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-home\") pod \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") "
Jul 28 01:40:02.763685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763660    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tls-certs\") pod \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\" (UID: \"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137\") "
Jul 28 01:40:02.763685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763675    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" (UID: "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:02.763853 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763789    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" (UID: "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:02.763896 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.763865    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-home" (OuterVolumeSpecName: "home") pod "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" (UID: "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:02.764034 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.764011    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-model-cache\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:02.764034 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.764032    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-home\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:02.764233 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.764041    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tmp-dir\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:02.765561 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.765530    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" (UID: "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:40:02.765659 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.765608    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-dshm" (OuterVolumeSpecName: "dshm") pod "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" (UID: "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:40:02.765659 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.765641    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-kube-api-access-sf57z" (OuterVolumeSpecName: "kube-api-access-sf57z") pod "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" (UID: "a6ebf3ed-6bd4-42eb-84dc-ce98d178f137"). InnerVolumeSpecName "kube-api-access-sf57z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:02.865262 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.865205    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-dshm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:02.865262 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.865257    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-tls-certs\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:02.865262 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:02.865271    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf57z\" (UniqueName: \"kubernetes.io/projected/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137-kube-api-access-sf57z\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:03.290664 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.290630    2576 generic.go:356] "Generic (PLEG): container finished" podID="a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" containerID="9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a" exitCode=0
Jul 28 01:40:03.290891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.290741    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"
Jul 28 01:40:03.290891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.290761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s" event={"ID":"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137","Type":"ContainerDied","Data":"9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a"}
Jul 28 01:40:03.290891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.290793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s" event={"ID":"a6ebf3ed-6bd4-42eb-84dc-ce98d178f137","Type":"ContainerDied","Data":"41e35c15cd0b770596111250e580b4b250ba932f92ec7c274fd53576fe3a4757"}
Jul 28 01:40:03.290891 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.290810    2576 scope.go:117] "RemoveContainer" containerID="9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a"
Jul 28 01:40:03.301089 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.301070    2576 scope.go:117] "RemoveContainer" containerID="9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a"
Jul 28 01:40:03.301405 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:40:03.301377    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a\": container with ID starting with 9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a not found: ID does not exist" containerID="9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a"
Jul 28 01:40:03.301497 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.301408    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a"} err="failed to get container status \"9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a\": rpc error: code = NotFound desc = could not find container \"9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a\": container with ID starting with 9dd2d778804aaaf35cc222918952eebfb2a7c83c0f0d2a3104f30ab8e0c4a38a not found: ID does not exist"
Jul 28 01:40:03.313717 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.313695    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"]
Jul 28 01:40:03.317406 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.317387    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c0266d/ipp-route-c0266d-kserve-5f7fcbb547-2wp9s"]
Jul 28 01:40:03.750854 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:03.750818    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" path="/var/lib/kubelet/pods/a6ebf3ed-6bd4-42eb-84dc-ce98d178f137/volumes"
Jul 28 01:40:24.061591 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:24.061559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:25.064109 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:25.064076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:31.416955 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.416864    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"]
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417244    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400488e7-c439-45e6-aa72-f4e28bcf3cdb" containerName="payload-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417256    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="400488e7-c439-45e6-aa72-f4e28bcf3cdb" containerName="payload-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417271    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc58796a-b674-47dc-bf92-e4bf933fbd30" containerName="payload-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417276    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc58796a-b674-47dc-bf92-e4bf933fbd30" containerName="payload-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417286    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" containerName="main"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417292    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" containerName="main"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417300    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b753e0d-b75f-4136-b5f2-cb98cefa579a" containerName="payload-pre-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417304    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b753e0d-b75f-4136-b5f2-cb98cefa579a" containerName="payload-pre-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417312    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e4109b-f81a-4be8-af77-606da8599936" containerName="payload-pre-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417317    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e4109b-f81a-4be8-af77-606da8599936" containerName="payload-pre-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417371    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b753e0d-b75f-4136-b5f2-cb98cefa579a" containerName="payload-pre-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417378    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc58796a-b674-47dc-bf92-e4bf933fbd30" containerName="payload-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417386    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e4109b-f81a-4be8-af77-606da8599936" containerName="payload-pre-processing"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417391    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6ebf3ed-6bd4-42eb-84dc-ce98d178f137" containerName="main"
Jul 28 01:40:31.417400 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.417398    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="400488e7-c439-45e6-aa72-f4e28bcf3cdb" containerName="payload-processing"
Jul 28 01:40:31.420376 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.420358    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:31.423272 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.423250    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:40:31.423403 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.423279    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:40:31.427338 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.427305    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"]
Jul 28 01:40:31.472802 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.472777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:31.516588 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.516563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52\" (UID: \"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:31.532288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.532260    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:31.618023 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.617987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52\" (UID: \"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:31.618638 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.618614    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52\" (UID: \"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:31.732159 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.732084    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:31.860704 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:31.860681    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"]
Jul 28 01:40:31.863443 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:40:31.863415    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44ceeb0_fa08_4f3b_9dab_8448b54c15b9.slice/crio-e91e9509279ebd4e9770550f0f18c8ce24aade5c54e59eec0a6551e51def1e34 WatchSource:0}: Error finding container e91e9509279ebd4e9770550f0f18c8ce24aade5c54e59eec0a6551e51def1e34: Status 404 returned error can't find the container with id e91e9509279ebd4e9770550f0f18c8ce24aade5c54e59eec0a6551e51def1e34
Jul 28 01:40:32.441287 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:32.441170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52" event={"ID":"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9","Type":"ContainerStarted","Data":"fd52d69bb4bf000a71027ba6ba5ac3a9c009d1399e3d8ca9b5e8a7f90e5fffc9"}
Jul 28 01:40:32.441287 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:32.441241    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52" event={"ID":"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9","Type":"ContainerStarted","Data":"e91e9509279ebd4e9770550f0f18c8ce24aade5c54e59eec0a6551e51def1e34"}
Jul 28 01:40:32.456971 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:32.456921    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52" podStartSLOduration=1.45690519 podStartE2EDuration="1.45690519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:40:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:40:32.456304214 +0000 UTC m=+3237.205442938" watchObservedRunningTime="2026-07-28 01:40:32.45690519 +0000 UTC m=+3237.206043947"
Jul 28 01:40:33.447652 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:33.447615    2576 generic.go:356] "Generic (PLEG): container finished" podID="c44ceeb0-fa08-4f3b-9dab-8448b54c15b9" containerID="fd52d69bb4bf000a71027ba6ba5ac3a9c009d1399e3d8ca9b5e8a7f90e5fffc9" exitCode=0
Jul 28 01:40:33.448035 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:33.447705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52" event={"ID":"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9","Type":"ContainerDied","Data":"fd52d69bb4bf000a71027ba6ba5ac3a9c009d1399e3d8ca9b5e8a7f90e5fffc9"}
Jul 28 01:40:34.588632 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:34.588607    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:34.645085 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:34.645050    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9-maas-api-service-ca\") pod \"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9\" (UID: \"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9\") "
Jul 28 01:40:34.645368 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:34.645344    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c44ceeb0-fa08-4f3b-9dab-8448b54c15b9" (UID: "c44ceeb0-fa08-4f3b-9dab-8448b54c15b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:34.746527 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:34.746464    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9-maas-api-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:35.458941 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:35.458916    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"
Jul 28 01:40:35.459118 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:35.458956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52" event={"ID":"c44ceeb0-fa08-4f3b-9dab-8448b54c15b9","Type":"ContainerDied","Data":"e91e9509279ebd4e9770550f0f18c8ce24aade5c54e59eec0a6551e51def1e34"}
Jul 28 01:40:35.459118 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:35.459001    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91e9509279ebd4e9770550f0f18c8ce24aade5c54e59eec0a6551e51def1e34"
Jul 28 01:40:42.554021 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.553983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:42.576571 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.576532    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"]
Jul 28 01:40:42.576820 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.576793    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r" podUID="a2928a61-07c4-4df6-803d-2936f8e2fb87" containerName="payload-processing" containerID="cri-o://5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94" gracePeriod=30
Jul 28 01:40:42.580917 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.580894    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"]
Jul 28 01:40:42.581842 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.581804    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd" podUID="7ff76ee4-28d8-4ea9-941e-2e748dfb3485" containerName="payload-pre-processing" containerID="cri-o://f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5" gracePeriod=30
Jul 28 01:40:42.654348 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.654317    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:42.717732 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.717696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:42.864515 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.864487    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:40:42.867241 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.867205    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:40:42.920812 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.920785    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-plugins-config-volume\") pod \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") "
Jul 28 01:40:42.920967 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.920822    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8w8wb\" (UniqueName: \"kubernetes.io/projected/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-kube-api-access-8w8wb\") pod \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\" (UID: \"7ff76ee4-28d8-4ea9-941e-2e748dfb3485\") "
Jul 28 01:40:42.920967 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.920842    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2928a61-07c4-4df6-803d-2936f8e2fb87-plugins-config-volume\") pod \"a2928a61-07c4-4df6-803d-2936f8e2fb87\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") "
Jul 28 01:40:42.920967 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.920889    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zd24v\" (UniqueName: \"kubernetes.io/projected/a2928a61-07c4-4df6-803d-2936f8e2fb87-kube-api-access-zd24v\") pod \"a2928a61-07c4-4df6-803d-2936f8e2fb87\" (UID: \"a2928a61-07c4-4df6-803d-2936f8e2fb87\") "
Jul 28 01:40:42.921207 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.921178    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ff76ee4-28d8-4ea9-941e-2e748dfb3485" (UID: "7ff76ee4-28d8-4ea9-941e-2e748dfb3485"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:42.921288 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.921256    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2928a61-07c4-4df6-803d-2936f8e2fb87-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a2928a61-07c4-4df6-803d-2936f8e2fb87" (UID: "a2928a61-07c4-4df6-803d-2936f8e2fb87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:42.922819 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.922801    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-kube-api-access-8w8wb" (OuterVolumeSpecName: "kube-api-access-8w8wb") pod "7ff76ee4-28d8-4ea9-941e-2e748dfb3485" (UID: "7ff76ee4-28d8-4ea9-941e-2e748dfb3485"). InnerVolumeSpecName "kube-api-access-8w8wb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:42.922921 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:42.922900    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2928a61-07c4-4df6-803d-2936f8e2fb87-kube-api-access-zd24v" (OuterVolumeSpecName: "kube-api-access-zd24v") pod "a2928a61-07c4-4df6-803d-2936f8e2fb87" (UID: "a2928a61-07c4-4df6-803d-2936f8e2fb87"). InnerVolumeSpecName "kube-api-access-zd24v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:40:43.021646 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.021621    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:43.021646 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.021642    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8wb\" (UniqueName: \"kubernetes.io/projected/7ff76ee4-28d8-4ea9-941e-2e748dfb3485-kube-api-access-8w8wb\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:43.021801 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.021653    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2928a61-07c4-4df6-803d-2936f8e2fb87-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:43.021801 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.021662    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zd24v\" (UniqueName: \"kubernetes.io/projected/a2928a61-07c4-4df6-803d-2936f8e2fb87-kube-api-access-zd24v\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:43.497407 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.497369    2576 generic.go:356] "Generic (PLEG): container finished" podID="7ff76ee4-28d8-4ea9-941e-2e748dfb3485" containerID="f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5" exitCode=0
Jul 28 01:40:43.497609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.497416    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"
Jul 28 01:40:43.497609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.497456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd" event={"ID":"7ff76ee4-28d8-4ea9-941e-2e748dfb3485","Type":"ContainerDied","Data":"f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5"}
Jul 28 01:40:43.497609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.497509    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd" event={"ID":"7ff76ee4-28d8-4ea9-941e-2e748dfb3485","Type":"ContainerDied","Data":"1929a91af7665a12678d4725e7e56738e8d8afd4c4cc028cb08a00a622e3f202"}
Jul 28 01:40:43.497609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.497536    2576 scope.go:117] "RemoveContainer" containerID="f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5"
Jul 28 01:40:43.504064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.499618    2576 generic.go:356] "Generic (PLEG): container finished" podID="a2928a61-07c4-4df6-803d-2936f8e2fb87" containerID="5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94" exitCode=0
Jul 28 01:40:43.504064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.499706    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r" event={"ID":"a2928a61-07c4-4df6-803d-2936f8e2fb87","Type":"ContainerDied","Data":"5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94"}
Jul 28 01:40:43.504064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.499780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r" event={"ID":"a2928a61-07c4-4df6-803d-2936f8e2fb87","Type":"ContainerDied","Data":"e38fad9b8c06d2a8bdf7de7fdbdabf436597e6eeb80828dc3d6c43b7a90cafea"}
Jul 28 01:40:43.504064 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.499871    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"
Jul 28 01:40:43.513802 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.513780    2576 scope.go:117] "RemoveContainer" containerID="f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5"
Jul 28 01:40:43.514076 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:40:43.514055    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5\": container with ID starting with f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5 not found: ID does not exist" containerID="f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5"
Jul 28 01:40:43.514134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.514083    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5"} err="failed to get container status \"f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5\": rpc error: code = NotFound desc = could not find container \"f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5\": container with ID starting with f20112cb5f90349d66969cdd3f1785f0346273823c1cf9f10c3ebdacb36f28f5 not found: ID does not exist"
Jul 28 01:40:43.514134 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.514099    2576 scope.go:117] "RemoveContainer" containerID="5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94"
Jul 28 01:40:43.521980 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.521959    2576 scope.go:117] "RemoveContainer" containerID="5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94"
Jul 28 01:40:43.522309 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:40:43.522290    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94\": container with ID starting with 5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94 not found: ID does not exist" containerID="5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94"
Jul 28 01:40:43.522388 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.522316    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94"} err="failed to get container status \"5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94\": rpc error: code = NotFound desc = could not find container \"5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94\": container with ID starting with 5e80e4a8d51746326d08528e521c82f27022c02277a2314189d47ed45eab6f94 not found: ID does not exist"
Jul 28 01:40:43.523430 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.523412    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"]
Jul 28 01:40:43.527129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.527110    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1ce59a-c6bc5b948-hnxqd"]
Jul 28 01:40:43.537216 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.537194    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"]
Jul 28 01:40:43.542875 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.542856    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1ce59a-7b8c867dcb-gzr7r"]
Jul 28 01:40:43.750271 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.750171    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff76ee4-28d8-4ea9-941e-2e748dfb3485" path="/var/lib/kubelet/pods/7ff76ee4-28d8-4ea9-941e-2e748dfb3485/volumes"
Jul 28 01:40:43.750619 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:43.750546    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2928a61-07c4-4df6-803d-2936f8e2fb87" path="/var/lib/kubelet/pods/a2928a61-07c4-4df6-803d-2936f8e2fb87/volumes"
Jul 28 01:40:46.481912 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:46.481883    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"]
Jul 28 01:40:46.484291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:46.484255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:46.484957 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:46.484933    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1ce59a-qtp52"]
Jul 28 01:40:46.868148 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:46.868120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:47.751020 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:47.750987    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c44ceeb0-fa08-4f3b-9dab-8448b54c15b9" path="/var/lib/kubelet/pods/c44ceeb0-fa08-4f3b-9dab-8448b54c15b9/volumes"
Jul 28 01:40:52.679448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:52.679417    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:52.742268 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:52.742218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:52.968869 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:52.968785    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:53.197612 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.197581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:40:53.349030 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.348990    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"]
Jul 28 01:40:53.349379 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349363    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2928a61-07c4-4df6-803d-2936f8e2fb87" containerName="payload-processing"
Jul 28 01:40:53.349379 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349378    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2928a61-07c4-4df6-803d-2936f8e2fb87" containerName="payload-processing"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349388    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff76ee4-28d8-4ea9-941e-2e748dfb3485" containerName="payload-pre-processing"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349394    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff76ee4-28d8-4ea9-941e-2e748dfb3485" containerName="payload-pre-processing"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349409    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c44ceeb0-fa08-4f3b-9dab-8448b54c15b9" containerName="revoke-keys"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349414    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c44ceeb0-fa08-4f3b-9dab-8448b54c15b9" containerName="revoke-keys"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349470    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2928a61-07c4-4df6-803d-2936f8e2fb87" containerName="payload-processing"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349481    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff76ee4-28d8-4ea9-941e-2e748dfb3485" containerName="payload-pre-processing"
Jul 28 01:40:53.349556 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.349488    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c44ceeb0-fa08-4f3b-9dab-8448b54c15b9" containerName="revoke-keys"
Jul 28 01:40:53.352518 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.352496    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:53.355324 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.355304    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxkp2\""
Jul 28 01:40:53.355426 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.355398    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 01:40:53.360581 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.360557    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"]
Jul 28 01:40:53.409695 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.409666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3833c-c9c6-46db-9de7-5f98a99959c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt\" (UID: \"8cb3833c-c9c6-46db-9de7-5f98a99959c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:53.510758 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.510735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3833c-c9c6-46db-9de7-5f98a99959c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt\" (UID: \"8cb3833c-c9c6-46db-9de7-5f98a99959c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:53.511304 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.511286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3833c-c9c6-46db-9de7-5f98a99959c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt\" (UID: \"8cb3833c-c9c6-46db-9de7-5f98a99959c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:53.664557 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.664498    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:53.805366 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:53.805206    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"]
Jul 28 01:40:53.808480 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:40:53.808454    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb3833c_c9c6_46db_9de7_5f98a99959c1.slice/crio-26a0d516c464bd96cec50d0115db98235d7f469108062d2fb9d9ad3e9e5d381a WatchSource:0}: Error finding container 26a0d516c464bd96cec50d0115db98235d7f469108062d2fb9d9ad3e9e5d381a: Status 404 returned error can't find the container with id 26a0d516c464bd96cec50d0115db98235d7f469108062d2fb9d9ad3e9e5d381a
Jul 28 01:40:54.552663 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:54.552485    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt" event={"ID":"8cb3833c-c9c6-46db-9de7-5f98a99959c1","Type":"ContainerStarted","Data":"96aca966cce6f8a9961da5b33eca1b75cbb24eb2a9a95d842a775ea1b12da4df"}
Jul 28 01:40:54.552663 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:54.552623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt" event={"ID":"8cb3833c-c9c6-46db-9de7-5f98a99959c1","Type":"ContainerStarted","Data":"26a0d516c464bd96cec50d0115db98235d7f469108062d2fb9d9ad3e9e5d381a"}
Jul 28 01:40:54.572273 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:54.572204    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt" podStartSLOduration=1.572186366 podStartE2EDuration="1.572186366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:40:54.569504424 +0000 UTC m=+3259.318643151" watchObservedRunningTime="2026-07-28 01:40:54.572186366 +0000 UTC m=+3259.321325092"
Jul 28 01:40:55.559129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:55.559096    2576 generic.go:356] "Generic (PLEG): container finished" podID="8cb3833c-c9c6-46db-9de7-5f98a99959c1" containerID="96aca966cce6f8a9961da5b33eca1b75cbb24eb2a9a95d842a775ea1b12da4df" exitCode=0
Jul 28 01:40:55.559543 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:55.559188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt" event={"ID":"8cb3833c-c9c6-46db-9de7-5f98a99959c1","Type":"ContainerDied","Data":"96aca966cce6f8a9961da5b33eca1b75cbb24eb2a9a95d842a775ea1b12da4df"}
Jul 28 01:40:56.694577 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:56.694555    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:56.839448 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:56.839376    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3833c-c9c6-46db-9de7-5f98a99959c1-maas-api-service-ca\") pod \"8cb3833c-c9c6-46db-9de7-5f98a99959c1\" (UID: \"8cb3833c-c9c6-46db-9de7-5f98a99959c1\") "
Jul 28 01:40:56.839791 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:56.839766    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb3833c-c9c6-46db-9de7-5f98a99959c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cb3833c-c9c6-46db-9de7-5f98a99959c1" (UID: "8cb3833c-c9c6-46db-9de7-5f98a99959c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:40:56.940205 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:56.940178    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb3833c-c9c6-46db-9de7-5f98a99959c1-maas-api-service-ca\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:40:57.570581 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:57.570552    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"
Jul 28 01:40:57.570753 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:57.570557    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt" event={"ID":"8cb3833c-c9c6-46db-9de7-5f98a99959c1","Type":"ContainerDied","Data":"26a0d516c464bd96cec50d0115db98235d7f469108062d2fb9d9ad3e9e5d381a"}
Jul 28 01:40:57.570753 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:40:57.570677    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26a0d516c464bd96cec50d0115db98235d7f469108062d2fb9d9ad3e9e5d381a"
Jul 28 01:41:03.751037 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.751010    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"]
Jul 28 01:41:03.751037 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.751044    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-14a1d0-29753370-lxhpm"]
Jul 28 01:41:03.762207 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.762178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:03.778831 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.778803    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"]
Jul 28 01:41:03.779051 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.779026    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2" podUID="c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" containerName="payload-processing" containerID="cri-o://c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248" gracePeriod=30
Jul 28 01:41:03.784398 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.783336    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"]
Jul 28 01:41:03.784398 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.783567    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx" podUID="ed127e22-f2df-44a2-b740-44184320c13d" containerName="payload-pre-processing" containerID="cri-o://e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262" gracePeriod=30
Jul 28 01:41:03.863090 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:03.863068    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:04.085359 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.085335    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:41:04.087990 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.087970    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:41:04.204545 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.204514    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjc8q\" (UniqueName: \"kubernetes.io/projected/ed127e22-f2df-44a2-b740-44184320c13d-kube-api-access-gjc8q\") pod \"ed127e22-f2df-44a2-b740-44184320c13d\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") "
Jul 28 01:41:04.204700 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.204551    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-plugins-config-volume\") pod \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") "
Jul 28 01:41:04.204700 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.204610    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6fkm\" (UniqueName: \"kubernetes.io/projected/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-kube-api-access-r6fkm\") pod \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\" (UID: \"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4\") "
Jul 28 01:41:04.204700 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.204660    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127e22-f2df-44a2-b740-44184320c13d-plugins-config-volume\") pod \"ed127e22-f2df-44a2-b740-44184320c13d\" (UID: \"ed127e22-f2df-44a2-b740-44184320c13d\") "
Jul 28 01:41:04.204989 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.204963    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" (UID: "c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:04.205098 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.205002    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed127e22-f2df-44a2-b740-44184320c13d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ed127e22-f2df-44a2-b740-44184320c13d" (UID: "ed127e22-f2df-44a2-b740-44184320c13d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:04.206677 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.206650    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed127e22-f2df-44a2-b740-44184320c13d-kube-api-access-gjc8q" (OuterVolumeSpecName: "kube-api-access-gjc8q") pod "ed127e22-f2df-44a2-b740-44184320c13d" (UID: "ed127e22-f2df-44a2-b740-44184320c13d"). InnerVolumeSpecName "kube-api-access-gjc8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:04.206768 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.206741    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-kube-api-access-r6fkm" (OuterVolumeSpecName: "kube-api-access-r6fkm") pod "c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" (UID: "c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4"). InnerVolumeSpecName "kube-api-access-r6fkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:04.305345 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.305288    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6fkm\" (UniqueName: \"kubernetes.io/projected/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-kube-api-access-r6fkm\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.305345 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.305310    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127e22-f2df-44a2-b740-44184320c13d-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.305345 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.305320    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjc8q\" (UniqueName: \"kubernetes.io/projected/ed127e22-f2df-44a2-b740-44184320c13d-kube-api-access-gjc8q\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.305345 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.305328    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:04.606663 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.606631    2576 generic.go:356] "Generic (PLEG): container finished" podID="c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" containerID="c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248" exitCode=0
Jul 28 01:41:04.606804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.606671    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"
Jul 28 01:41:04.606804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.606702    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2" event={"ID":"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4","Type":"ContainerDied","Data":"c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248"}
Jul 28 01:41:04.606804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.606747    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2" event={"ID":"c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4","Type":"ContainerDied","Data":"d04517db86de5ee46417d1e8f81b50fed0cac2af4aa577a2eff653097d368f24"}
Jul 28 01:41:04.606804 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.606771    2576 scope.go:117] "RemoveContainer" containerID="c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248"
Jul 28 01:41:04.608052 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.608027    2576 generic.go:356] "Generic (PLEG): container finished" podID="ed127e22-f2df-44a2-b740-44184320c13d" containerID="e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262" exitCode=0
Jul 28 01:41:04.608174 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.608088    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"
Jul 28 01:41:04.608174 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.608108    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx" event={"ID":"ed127e22-f2df-44a2-b740-44184320c13d","Type":"ContainerDied","Data":"e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262"}
Jul 28 01:41:04.608174 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.608160    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx" event={"ID":"ed127e22-f2df-44a2-b740-44184320c13d","Type":"ContainerDied","Data":"093e4a0405ffbba1e59a381eff1a2dd31599f04a39a18f250e407e8414c016cb"}
Jul 28 01:41:04.616292 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.616244    2576 scope.go:117] "RemoveContainer" containerID="c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248"
Jul 28 01:41:04.616532 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:41:04.616508    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248\": container with ID starting with c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248 not found: ID does not exist" containerID="c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248"
Jul 28 01:41:04.616609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.616541    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248"} err="failed to get container status \"c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248\": rpc error: code = NotFound desc = could not find container \"c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248\": container with ID starting with c0105050d2da0418488d12e0e279a7033582c3985bf8feeea840a370165cc248 not found: ID does not exist"
Jul 28 01:41:04.616609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.616563    2576 scope.go:117] "RemoveContainer" containerID="e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262"
Jul 28 01:41:04.623992 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.623977    2576 scope.go:117] "RemoveContainer" containerID="e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262"
Jul 28 01:41:04.624246 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:41:04.624212    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262\": container with ID starting with e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262 not found: ID does not exist" containerID="e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262"
Jul 28 01:41:04.624299 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.624253    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262"} err="failed to get container status \"e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262\": rpc error: code = NotFound desc = could not find container \"e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262\": container with ID starting with e33f3fc3db08214edcfa6d723662bc5fa1a74a8038062a0d1540c140b56db262 not found: ID does not exist"
Jul 28 01:41:04.628884 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.628863    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"]
Jul 28 01:41:04.637419 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.637395    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-14a1d0-5c4bc8d69d-nwjl2"]
Jul 28 01:41:04.646766 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.646747    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"]
Jul 28 01:41:04.649816 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:04.649796    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-14a1d0-5b66896cb4-qf6lx"]
Jul 28 01:41:05.751434 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:05.751400    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e3dd7fa-6317-4dfd-9d86-558913e31a66" path="/var/lib/kubelet/pods/1e3dd7fa-6317-4dfd-9d86-558913e31a66/volumes"
Jul 28 01:41:05.751806 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:05.751752    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" path="/var/lib/kubelet/pods/c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4/volumes"
Jul 28 01:41:05.752067 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:05.752053    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed127e22-f2df-44a2-b740-44184320c13d" path="/var/lib/kubelet/pods/ed127e22-f2df-44a2-b740-44184320c13d/volumes"
Jul 28 01:41:08.448852 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:08.448819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:08.454343 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:08.454289    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"]
Jul 28 01:41:08.457291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:08.457267    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-14a1d0-5z4qt"]
Jul 28 01:41:08.868293 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:08.868263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:09.752626 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:09.752590    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb3833c-c9c6-46db-9de7-5f98a99959c1" path="/var/lib/kubelet/pods/8cb3833c-c9c6-46db-9de7-5f98a99959c1/volumes"
Jul 28 01:41:14.663184 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:14.663144    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:14.721778 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:14.721752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:14.913945 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:14.913865    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:15.148045 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:15.148012    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:25.666570 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.666539    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"]
Jul 28 01:41:25.671826 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.671777    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:25.672161 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.672142    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-3131e3-29753370-rqr8k"]
Jul 28 01:41:25.698861 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.698831    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"]
Jul 28 01:41:25.699129 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.699085    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v" podUID="820afed0-72bd-42e0-922a-2af06a4fc493" containerName="payload-processing" containerID="cri-o://457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8" gracePeriod=30
Jul 28 01:41:25.704585 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.704547    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"]
Jul 28 01:41:25.704958 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.704786    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc" podUID="65f8d5ff-21fe-421c-8f3b-d16b4d90e891" containerName="payload-pre-processing" containerID="cri-o://18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb" gracePeriod=30
Jul 28 01:41:25.736238 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.736169    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:25.750952 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.750921    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9df4025b-7993-439a-b856-447d971728ec" path="/var/lib/kubelet/pods/9df4025b-7993-439a-b856-447d971728ec/volumes"
Jul 28 01:41:25.982046 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.982025    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:41:25.984609 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:25.984592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:41:26.082035 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.081998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:26.087065 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.087046    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-plugins-config-volume\") pod \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") "
Jul 28 01:41:26.087131 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.087111    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7mfq\" (UniqueName: \"kubernetes.io/projected/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-kube-api-access-j7mfq\") pod \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\" (UID: \"65f8d5ff-21fe-421c-8f3b-d16b4d90e891\") "
Jul 28 01:41:26.087178 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.087161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmfbw\" (UniqueName: \"kubernetes.io/projected/820afed0-72bd-42e0-922a-2af06a4fc493-kube-api-access-jmfbw\") pod \"820afed0-72bd-42e0-922a-2af06a4fc493\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") "
Jul 28 01:41:26.087252 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.087185    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820afed0-72bd-42e0-922a-2af06a4fc493-plugins-config-volume\") pod \"820afed0-72bd-42e0-922a-2af06a4fc493\" (UID: \"820afed0-72bd-42e0-922a-2af06a4fc493\") "
Jul 28 01:41:26.087565 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.087536    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65f8d5ff-21fe-421c-8f3b-d16b4d90e891" (UID: "65f8d5ff-21fe-421c-8f3b-d16b4d90e891"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:26.087654 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.087566    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820afed0-72bd-42e0-922a-2af06a4fc493-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "820afed0-72bd-42e0-922a-2af06a4fc493" (UID: "820afed0-72bd-42e0-922a-2af06a4fc493"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:41:26.089200 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.089175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820afed0-72bd-42e0-922a-2af06a4fc493-kube-api-access-jmfbw" (OuterVolumeSpecName: "kube-api-access-jmfbw") pod "820afed0-72bd-42e0-922a-2af06a4fc493" (UID: "820afed0-72bd-42e0-922a-2af06a4fc493"). InnerVolumeSpecName "kube-api-access-jmfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:26.089291 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.089193    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-kube-api-access-j7mfq" (OuterVolumeSpecName: "kube-api-access-j7mfq") pod "65f8d5ff-21fe-421c-8f3b-d16b4d90e891" (UID: "65f8d5ff-21fe-421c-8f3b-d16b4d90e891"). InnerVolumeSpecName "kube-api-access-j7mfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:41:26.188537 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.188504    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfbw\" (UniqueName: \"kubernetes.io/projected/820afed0-72bd-42e0-922a-2af06a4fc493-kube-api-access-jmfbw\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.188537 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.188532    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820afed0-72bd-42e0-922a-2af06a4fc493-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.188704 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.188545    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-plugins-config-volume\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.188704 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.188558    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7mfq\" (UniqueName: \"kubernetes.io/projected/65f8d5ff-21fe-421c-8f3b-d16b4d90e891-kube-api-access-j7mfq\") on node \"ip-10-0-140-154.ec2.internal\" DevicePath \"\""
Jul 28 01:41:26.720174 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.720137    2576 generic.go:356] "Generic (PLEG): container finished" podID="820afed0-72bd-42e0-922a-2af06a4fc493" containerID="457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8" exitCode=0
Jul 28 01:41:26.720624 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.720184    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"
Jul 28 01:41:26.720624 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.720243    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v" event={"ID":"820afed0-72bd-42e0-922a-2af06a4fc493","Type":"ContainerDied","Data":"457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8"}
Jul 28 01:41:26.720624 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.720308    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v" event={"ID":"820afed0-72bd-42e0-922a-2af06a4fc493","Type":"ContainerDied","Data":"fd3467dbb6f8eb72b9118035b51b7a47aba0da1a1dc89d02fcb0aca630021d0c"}
Jul 28 01:41:26.720624 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.720343    2576 scope.go:117] "RemoveContainer" containerID="457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8"
Jul 28 01:41:26.721787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.721614    2576 generic.go:356] "Generic (PLEG): container finished" podID="65f8d5ff-21fe-421c-8f3b-d16b4d90e891" containerID="18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb" exitCode=0
Jul 28 01:41:26.721787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.721669    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc" event={"ID":"65f8d5ff-21fe-421c-8f3b-d16b4d90e891","Type":"ContainerDied","Data":"18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb"}
Jul 28 01:41:26.721787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.721711    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc" event={"ID":"65f8d5ff-21fe-421c-8f3b-d16b4d90e891","Type":"ContainerDied","Data":"c1413c40b373cff20d79fd885124ff80093a851085c48030b60a5349075d512d"}
Jul 28 01:41:26.721787 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.721725    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"
Jul 28 01:41:26.731105 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.731088    2576 scope.go:117] "RemoveContainer" containerID="457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8"
Jul 28 01:41:26.731482 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:41:26.731452    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8\": container with ID starting with 457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8 not found: ID does not exist" containerID="457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8"
Jul 28 01:41:26.731564 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.731495    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8"} err="failed to get container status \"457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8\": rpc error: code = NotFound desc = could not find container \"457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8\": container with ID starting with 457639a91c595176c6a9c4b9112357a6af0b824b5067273138459f632c711ce8 not found: ID does not exist"
Jul 28 01:41:26.731564 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.731516    2576 scope.go:117] "RemoveContainer" containerID="18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb"
Jul 28 01:41:26.738997 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.738978    2576 scope.go:117] "RemoveContainer" containerID="18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb"
Jul 28 01:41:26.739237 ip-10-0-140-154 kubenswrapper[2576]: E0728 01:41:26.739207    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb\": container with ID starting with 18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb not found: ID does not exist" containerID="18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb"
Jul 28 01:41:26.739302 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.739243    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb"} err="failed to get container status \"18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb\": rpc error: code = NotFound desc = could not find container \"18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb\": container with ID starting with 18bc6c6e6d51394c54a0c581b95e0b367e680cb5797b70fcf322562341eda3cb not found: ID does not exist"
Jul 28 01:41:26.742664 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.742642    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"]
Jul 28 01:41:26.746303 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.746282    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-3131e3-7f59bbc7d6-vn55v"]
Jul 28 01:41:26.756734 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.756710    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"]
Jul 28 01:41:26.760495 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:26.760475    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-3131e3-86b87dcd44-9k7zc"]
Jul 28 01:41:27.750214 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:27.750183    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65f8d5ff-21fe-421c-8f3b-d16b4d90e891" path="/var/lib/kubelet/pods/65f8d5ff-21fe-421c-8f3b-d16b4d90e891/volumes"
Jul 28 01:41:27.750600 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:27.750571    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="820afed0-72bd-42e0-922a-2af06a4fc493" path="/var/lib/kubelet/pods/820afed0-72bd-42e0-922a-2af06a4fc493/volumes"
Jul 28 01:41:30.363433 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:30.363396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:30.719091 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:30.719014    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:36.111616 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:36.111584    2576 scope.go:117] "RemoveContainer" containerID="c4ce90d24f630546a983050a90822cb3fbb1228be6fa073fd8cfd72402000014"
Jul 28 01:41:36.119201 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:36.119183    2576 scope.go:117] "RemoveContainer" containerID="a5344b69f9e98e39abbe3df9f259211de1650c60f0df753cd3d40d240027dfd9"
Jul 28 01:41:36.519575 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:36.519489    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:36.627127 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:36.627087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:36.866206 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:36.866170    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:37.023818 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:37.023787    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:42.875399 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:42.875370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rxsnd_cf6ef881-3c12-45bd-b734-a5e2833988e9/manager/0.log"
Jul 28 01:41:43.734701 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:43.734670    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-mtj7j_e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050/copy-manifests/0.log"
Jul 28 01:41:43.741685 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:43.741657    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-mtj7j_e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050/manager/0.log"
Jul 28 01:41:44.449748 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:44.449724    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753370-7fdzm_cb5d3941-844d-4964-866b-366c1ad6d34c/cleanup/0.log"
Jul 28 01:41:46.449480 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:46.449447    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k7jm2_3f149aef-5e7c-41ea-9860-14a8863a1adf/manager/0.log"
Jul 28 01:41:47.400179 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:47.400144    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-jzsjp_625d301d-b05b-47f3-8fa5-c808343fa5b3/kube-auth-proxy/0.log"
Jul 28 01:41:47.753473 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:47.753394    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vtbwh_7041fa74-1187-4ba0-aa28-1405298481c6/payload-processing/0.log"
Jul 28 01:41:48.348316 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:48.348289    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-pxtw6_7eaad0c1-1a15-4729-b3d8-d59f43ab7d60/main/0.log"
Jul 28 01:41:48.683118 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:48.683045    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-wmbxr_f247398a-2c9a-4ff3-a791-fbc8bbf7e882/main/0.log"
Jul 28 01:41:48.797637 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:48.797610    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-42t64_bc67092a-61b5-44f5-b57a-6df58148de2e/main/0.log"
Jul 28 01:41:54.824427 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824389    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2v9r2/must-gather-z5krd"]
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824690    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb3833c-c9c6-46db-9de7-5f98a99959c1" containerName="revoke-keys"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824701    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb3833c-c9c6-46db-9de7-5f98a99959c1" containerName="revoke-keys"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824730    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed127e22-f2df-44a2-b740-44184320c13d" containerName="payload-pre-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824736    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed127e22-f2df-44a2-b740-44184320c13d" containerName="payload-pre-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824743    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="820afed0-72bd-42e0-922a-2af06a4fc493" containerName="payload-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824749    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="820afed0-72bd-42e0-922a-2af06a4fc493" containerName="payload-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824757    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65f8d5ff-21fe-421c-8f3b-d16b4d90e891" containerName="payload-pre-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824762    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="65f8d5ff-21fe-421c-8f3b-d16b4d90e891" containerName="payload-pre-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824773    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" containerName="payload-processing"
Jul 28 01:41:54.824811 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824778    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" containerName="payload-processing"
Jul 28 01:41:54.825420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824826    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="65f8d5ff-21fe-421c-8f3b-d16b4d90e891" containerName="payload-pre-processing"
Jul 28 01:41:54.825420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824833    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed127e22-f2df-44a2-b740-44184320c13d" containerName="payload-pre-processing"
Jul 28 01:41:54.825420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824839    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c85c5ea7-6b2d-4d0b-a1f4-a17143a62bb4" containerName="payload-processing"
Jul 28 01:41:54.825420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824844    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb3833c-c9c6-46db-9de7-5f98a99959c1" containerName="revoke-keys"
Jul 28 01:41:54.825420 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.824851    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="820afed0-72bd-42e0-922a-2af06a4fc493" containerName="payload-processing"
Jul 28 01:41:54.827897 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.827875    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:54.832179 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.832157    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2v9r2\"/\"kube-root-ca.crt\""
Jul 28 01:41:54.832459 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.832443    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2v9r2\"/\"openshift-service-ca.crt\""
Jul 28 01:41:54.841107 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.841086    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/must-gather-z5krd"]
Jul 28 01:41:54.855050 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.855028    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:54.931160 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.931130    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:41:54.935279 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.935258    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hblm\" (UniqueName: \"kubernetes.io/projected/64ca6f53-da6b-4e3d-aa22-ad71aa10404a-kube-api-access-4hblm\") pod \"must-gather-z5krd\" (UID: \"64ca6f53-da6b-4e3d-aa22-ad71aa10404a\") " pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:54.935360 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:54.935305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64ca6f53-da6b-4e3d-aa22-ad71aa10404a-must-gather-output\") pod \"must-gather-z5krd\" (UID: \"64ca6f53-da6b-4e3d-aa22-ad71aa10404a\") " pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:55.036469 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.036434    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hblm\" (UniqueName: \"kubernetes.io/projected/64ca6f53-da6b-4e3d-aa22-ad71aa10404a-kube-api-access-4hblm\") pod \"must-gather-z5krd\" (UID: \"64ca6f53-da6b-4e3d-aa22-ad71aa10404a\") " pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:55.036624 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.036478    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64ca6f53-da6b-4e3d-aa22-ad71aa10404a-must-gather-output\") pod \"must-gather-z5krd\" (UID: \"64ca6f53-da6b-4e3d-aa22-ad71aa10404a\") " pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:55.036847 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.036827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/64ca6f53-da6b-4e3d-aa22-ad71aa10404a-must-gather-output\") pod \"must-gather-z5krd\" (UID: \"64ca6f53-da6b-4e3d-aa22-ad71aa10404a\") " pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:55.045015 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.044991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hblm\" (UniqueName: \"kubernetes.io/projected/64ca6f53-da6b-4e3d-aa22-ad71aa10404a-kube-api-access-4hblm\") pod \"must-gather-z5krd\" (UID: \"64ca6f53-da6b-4e3d-aa22-ad71aa10404a\") " pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:55.136739 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.136661    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/must-gather-z5krd"
Jul 28 01:41:55.270341 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.270301    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/must-gather-z5krd"]
Jul 28 01:41:55.273116 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:41:55.273091    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ca6f53_da6b_4e3d_aa22_ad71aa10404a.slice/crio-5a2327e38f623a11f353077edd89edb33d360310b32ab92638645d60d5cb9a75 WatchSource:0}: Error finding container 5a2327e38f623a11f353077edd89edb33d360310b32ab92638645d60d5cb9a75: Status 404 returned error can't find the container with id 5a2327e38f623a11f353077edd89edb33d360310b32ab92638645d60d5cb9a75
Jul 28 01:41:55.274785 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.274768    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:41:55.854454 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:55.854413    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/must-gather-z5krd" event={"ID":"64ca6f53-da6b-4e3d-aa22-ad71aa10404a","Type":"ContainerStarted","Data":"5a2327e38f623a11f353077edd89edb33d360310b32ab92638645d60d5cb9a75"}
Jul 28 01:41:56.861927 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:56.861890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/must-gather-z5krd" event={"ID":"64ca6f53-da6b-4e3d-aa22-ad71aa10404a","Type":"ContainerStarted","Data":"282ce0eadfd6df7804eac76cccd29c95ac97eed5af95f64cb3ecb96ceb50c577"}
Jul 28 01:41:56.862542 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:56.861936    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/must-gather-z5krd" event={"ID":"64ca6f53-da6b-4e3d-aa22-ad71aa10404a","Type":"ContainerStarted","Data":"76e31241c2587ae9688770b86bb9504933d0da8b3dafdaa85e51e6988b2394a3"}
Jul 28 01:41:56.881399 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:56.881341    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2v9r2/must-gather-z5krd" podStartSLOduration=2.164785687 podStartE2EDuration="2.881319191s" totalImagesPullingTime="716.533504ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:41:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:41:56.87832327 +0000 UTC m=+3321.627461998" watchObservedRunningTime="2026-07-28 01:41:56.881319191 +0000 UTC m=+3321.630457916"
Jul 28 01:41:57.578308 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:57.578270    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zw59q_5cd31cd7-5079-487f-9a62-06a8854a3f58/global-pull-secret-syncer/0.log"
Jul 28 01:41:57.630062 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:57.630027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log"
Jul 28 01:41:57.764301 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:41:57.764266    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-154.ec2.internal_851ad20362c20c22501be689ae7a68aa/haproxy/0.log"
Jul 28 01:42:03.441247 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:03.440466    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-k7jm2_3f149aef-5e7c-41ea-9860-14a8863a1adf/manager/0.log"
Jul 28 01:42:05.350503 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.350470    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vvhtb_6dc160e0-e027-460b-b5a0-ccd194d013a6/kube-state-metrics/0.log"
Jul 28 01:42:05.371344 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.371295    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vvhtb_6dc160e0-e027-460b-b5a0-ccd194d013a6/kube-rbac-proxy-main/0.log"
Jul 28 01:42:05.396142 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.396110    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-vvhtb_6dc160e0-e027-460b-b5a0-ccd194d013a6/kube-rbac-proxy-self/0.log"
Jul 28 01:42:05.422749 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.422721    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-ff65875f-tlftv_ad58434d-62ed-4820-bd0b-71f91dfd24e6/metrics-server/0.log"
Jul 28 01:42:05.554546 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.554467    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqgsp_0f59960c-15ec-4904-ade3-e51e6e503ede/node-exporter/0.log"
Jul 28 01:42:05.575008 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.574982    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqgsp_0f59960c-15ec-4904-ade3-e51e6e503ede/kube-rbac-proxy/0.log"
Jul 28 01:42:05.596088 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.596063    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sqgsp_0f59960c-15ec-4904-ade3-e51e6e503ede/init-textfile/0.log"
Jul 28 01:42:05.708847 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.708814    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mchrn_5020ccf5-e631-4e00-8445-11a51559a7e4/kube-rbac-proxy-main/0.log"
Jul 28 01:42:05.730077 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.730027    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mchrn_5020ccf5-e631-4e00-8445-11a51559a7e4/kube-rbac-proxy-self/0.log"
Jul 28 01:42:05.751733 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.751703    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mchrn_5020ccf5-e631-4e00-8445-11a51559a7e4/openshift-state-metrics/0.log"
Jul 28 01:42:05.807154 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.807124    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/prometheus/0.log"
Jul 28 01:42:05.832473 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.832437    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/config-reloader/0.log"
Jul 28 01:42:05.855182 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.855148    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/thanos-sidecar/0.log"
Jul 28 01:42:05.878836 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.878808    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/kube-rbac-proxy-web/0.log"
Jul 28 01:42:05.900697 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.900670    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/kube-rbac-proxy/0.log"
Jul 28 01:42:05.922629 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.922599    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/kube-rbac-proxy-thanos/0.log"
Jul 28 01:42:05.946952 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:05.946925    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_1247c28d-cab7-4e1e-88ed-39bcde73636a/init-config-reloader/0.log"
Jul 28 01:42:06.023099 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:06.023002    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-v5zmn_6f4e0fa5-39b3-4c7c-89d6-b35e5d703969/prometheus-operator-admission-webhook/0.log"
Jul 28 01:42:07.607763 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.607727    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"]
Jul 28 01:42:07.616430 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.616396    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.619754 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.619729    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2v9r2\"/\"default-dockercfg-d6gzj\""
Jul 28 01:42:07.623117 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.623093    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"]
Jul 28 01:42:07.763642 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.763607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-sys\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.763846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.763668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-proc\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.763846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.763711    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-podres\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.763846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.763754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-lib-modules\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.763846 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.763796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlcf\" (UniqueName: \"kubernetes.io/projected/c6a227bd-bd83-4442-80f8-8932f92e80f9-kube-api-access-vnlcf\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.864551 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.864469    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-sys\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.864551 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.864537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-proc\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.864773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.864578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-podres\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.864773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.864625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-lib-modules\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.864773 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.864725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-proc\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.864939 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.864804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlcf\" (UniqueName: \"kubernetes.io/projected/c6a227bd-bd83-4442-80f8-8932f92e80f9-kube-api-access-vnlcf\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.865070 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.865043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-sys\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.865258 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.865235    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-lib-modules\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.865368 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.865140    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c6a227bd-bd83-4442-80f8-8932f92e80f9-podres\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.873951 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.873931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlcf\" (UniqueName: \"kubernetes.io/projected/c6a227bd-bd83-4442-80f8-8932f92e80f9-kube-api-access-vnlcf\") pod \"perf-node-gather-daemonset-pplss\" (UID: \"c6a227bd-bd83-4442-80f8-8932f92e80f9\") " pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:07.929961 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:07.929922    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:08.110572 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:08.110542    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"]
Jul 28 01:42:08.115825 ip-10-0-140-154 kubenswrapper[2576]: W0728 01:42:08.115733    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc6a227bd_bd83_4442_80f8_8932f92e80f9.slice/crio-e6468b63952dc630336d5c649fd7e92ba592b081906a7200a1ca3878ded19f9f WatchSource:0}: Error finding container e6468b63952dc630336d5c649fd7e92ba592b081906a7200a1ca3878ded19f9f: Status 404 returned error can't find the container with id e6468b63952dc630336d5c649fd7e92ba592b081906a7200a1ca3878ded19f9f
Jul 28 01:42:08.588457 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:08.588420    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6644d58899-bzjfq_b58d20e0-bedd-45ca-a0ad-baa2893b0f42/console/0.log"
Jul 28 01:42:08.946750 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:08.946662    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss" event={"ID":"c6a227bd-bd83-4442-80f8-8932f92e80f9","Type":"ContainerStarted","Data":"5dee9919743108ec1d13b2020e1b60e959ddaaee60027422f2429f3d0780786b"}
Jul 28 01:42:08.946750 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:08.946745    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss" event={"ID":"c6a227bd-bd83-4442-80f8-8932f92e80f9","Type":"ContainerStarted","Data":"e6468b63952dc630336d5c649fd7e92ba592b081906a7200a1ca3878ded19f9f"}
Jul 28 01:42:08.947181 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:08.946790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:08.965993 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:08.965941    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss" podStartSLOduration=1.9659240599999999 podStartE2EDuration="1.96592406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:42:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:42:08.96455639 +0000 UTC m=+3333.713695115" watchObservedRunningTime="2026-07-28 01:42:08.96592406 +0000 UTC m=+3333.715062786"
Jul 28 01:42:10.266281 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:10.266239    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2nsh7_7adf268e-0c53-4229-8416-847cffcc6dc2/dns/0.log"
Jul 28 01:42:10.300786 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:10.300723    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-2nsh7_7adf268e-0c53-4229-8416-847cffcc6dc2/kube-rbac-proxy/0.log"
Jul 28 01:42:10.502201 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:10.502170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z427m_692b3348-2671-407a-9d86-fa70cdf82395/dns-node-resolver/0.log"
Jul 28 01:42:11.051592 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:11.051545    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2575g_d3121109-447e-4c0e-a7d5-1a312109d5ba/node-ca/0.log"
Jul 28 01:42:12.053829 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:12.053770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:42:12.127638 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:12.127605    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-kmw85_8a3fb96e-52a8-4ebd-9333-7850f1805504/konnectivity-agent/0.log" limit=1024
Jul 28 01:42:12.186454 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:12.186423    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-jzsjp_625d301d-b05b-47f3-8fa5-c808343fa5b3/kube-auth-proxy/0.log"
Jul 28 01:42:12.306544 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:12.306516    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vtbwh_7041fa74-1187-4ba0-aa28-1405298481c6/payload-processing/0.log"
Jul 28 01:42:13.107690 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:13.107664    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mlmxc_7089b41c-0d46-4691-bb98-b3f46dd0887f/serve-healthcheck-canary/0.log"
Jul 28 01:42:13.607744 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:13.607716    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzfc2_a695d7f7-26eb-4dce-82f7-7b3020213bed/kube-rbac-proxy/0.log"
Jul 28 01:42:13.627044 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:13.627018    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzfc2_a695d7f7-26eb-4dce-82f7-7b3020213bed/exporter/0.log"
Jul 28 01:42:13.646965 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:13.646941    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-bzfc2_a695d7f7-26eb-4dce-82f7-7b3020213bed/extractor/0.log"
Jul 28 01:42:14.966056 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:14.965161    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2v9r2/perf-node-gather-daemonset-pplss"
Jul 28 01:42:15.900474 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:15.900442    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-rxsnd_cf6ef881-3c12-45bd-b734-a5e2833988e9/manager/0.log"
Jul 28 01:42:16.547534 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:16.547505    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-mtj7j_e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050/manager/0.log"
Jul 28 01:42:16.569841 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:16.569812    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-644f76477d-mtj7j_e1517e3f-f9a8-4d3e-b1d3-3d4f92ba4050/copy-manifests/0.log"
Jul 28 01:42:17.933979 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:17.933937    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-jzsjp_625d301d-b05b-47f3-8fa5-c808343fa5b3/kube-auth-proxy/0.log"
Jul 28 01:42:17.985255 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:17.985127    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f79f7c78b-kx45j_6e8d2ecc-bba7-4358-8530-bc9045a945eb/manager/0.log"
Jul 28 01:42:18.035422 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:18.035396    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vtbwh_7041fa74-1187-4ba0-aa28-1405298481c6/payload-processing/0.log"
Jul 28 01:42:19.663626 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:19.663529    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-c44798b85-jzsjp_625d301d-b05b-47f3-8fa5-c808343fa5b3/kube-auth-proxy/0.log"
Jul 28 01:42:19.766535 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:19.766505    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-vtbwh_7041fa74-1187-4ba0-aa28-1405298481c6/payload-processing/0.log"
Jul 28 01:42:24.101715 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.101687    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-2fvjp_05561ac3-c638-4293-9448-cadf817e389a/kube-multus/0.log"
Jul 28 01:42:24.335063 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.335038    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/kube-multus-additional-cni-plugins/0.log"
Jul 28 01:42:24.358246 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.358170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/egress-router-binary-copy/0.log"
Jul 28 01:42:24.379947 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.379923    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/cni-plugins/0.log"
Jul 28 01:42:24.402227 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.402208    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/bond-cni-plugin/0.log"
Jul 28 01:42:24.423711 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.423688    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/routeoverride-cni/0.log"
Jul 28 01:42:24.446122 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.446103    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/whereabouts-cni-bincopy/0.log"
Jul 28 01:42:24.466572 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.466544    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frfxj_2c427abf-95df-4799-a719-3d13f91bd5eb/whereabouts-cni/0.log"
Jul 28 01:42:24.676608 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.676532    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f4x8d_3c50f947-74a7-4285-adaa-0e77f24fc849/network-metrics-daemon/0.log"
Jul 28 01:42:24.691000 ip-10-0-140-154 kubenswrapper[2576]: I0728 01:42:24.690958    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-f4x8d_3c50f947-74a7-4285-adaa-0e77f24fc849/kube-rbac-proxy/0.log"