Jul 28 10:21:17.289207 ip-10-0-131-67 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 10:21:17.289218 ip-10-0-131-67 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 10:21:17.289227 ip-10-0-131-67 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 10:21:17.289565 ip-10-0-131-67 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 10:21:27.482447 ip-10-0-131-67 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 10:21:27.482468 ip-10-0-131-67 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 864d9620df8a4df7b11f035911f53cb5 --
Jul 28 10:23:28.848605 ip-10-0-131-67 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 10:23:29.220680 ip-10-0-131-67 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:23:29.220680 ip-10-0-131-67 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 10:23:29.220680 ip-10-0-131-67 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:23:29.220680 ip-10-0-131-67 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 10:23:29.220680 ip-10-0-131-67 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:23:29.221906 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.221517    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 10:23:29.228413 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228384    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:23:29.228413 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228407    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:23:29.228413 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228413    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:23:29.228413 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228417    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:23:29.228413 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228420    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:23:29.228413 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228423    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228426    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228429    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228433    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228436    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228439    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228442    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228445    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228447    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228450    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228453    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228456    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228458    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228462    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228464    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228469    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228474    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228477    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228480    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:23:29.228664 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228485    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228495    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228500    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228503    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228505    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228508    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228510    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228513    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228516    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228518    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228521    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228524    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228527    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228530    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228532    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228535    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228538    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228540    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228544    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:23:29.229115 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228546    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228549    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228551    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228554    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228556    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228559    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228561    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228565    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228568    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228570    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228573    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228576    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228578    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228581    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228583    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228585    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228589    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228592    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228596    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228599    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:23:29.229597 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228601    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228603    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228606    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228609    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228612    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228614    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228617    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228619    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228623    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228625    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228628    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228630    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228633    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228636    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228638    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228641    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228643    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228646    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228648    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228650    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:23:29.230069 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228654    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228656    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228659    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228662    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228664    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228667    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228669    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228672    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228675    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228678    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228680    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228683    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228685    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228688    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228691    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.228694    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229468    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229478    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229481    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:23:29.230565 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229484    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229487    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229491    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229494    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229497    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229499    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229502    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229505    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229508    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229511    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229513    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229516    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229518    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229521    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229524    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229526    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229529    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229531    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229534    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229537    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:23:29.231058 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229539    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229542    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229546    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229550    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229553    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229556    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229559    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229563    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229567    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229570    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229572    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229575    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229578    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229581    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229583    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229586    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229589    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229592    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229595    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229598    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:23:29.231568 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229601    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229604    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229606    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229609    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229612    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229614    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229617    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229620    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229623    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229626    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229629    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229632    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229634    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229636    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229639    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229641    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229645    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229648    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229650    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229653    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:23:29.232051 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229656    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229659    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229661    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229664    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229666    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229669    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229671    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229674    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229676    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229679    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229681    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229684    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229686    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229689    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229691    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229694    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229696    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229699    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229701    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229703    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:23:29.232608 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229709    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229712    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229714    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229717    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229720    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229722    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229725    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229727    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229730    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229732    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229735    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229738    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229741    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229744    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229746    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.229749    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230149    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230155    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230159    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:23:29.233101 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230163    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230165    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230168    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230171    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230174    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230178    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230181    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230184    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230187    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230190    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230193    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230195    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230198    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230210    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230213    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230217    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230219    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230222    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230224    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:23:29.233569 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230227    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230230    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230232    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230235    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230237    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230240    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230243    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230246    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230248    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230251    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230253    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230256    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230258    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230261    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230263    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230283    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230287    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230291    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230296    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230299    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:23:29.234034 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230302    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230304    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230307    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230309    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230312    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230314    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230317    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230322    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230324    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230327    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230329    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230332    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230334    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230337    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230339    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230342    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230344    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230346    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230349    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230352    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230354    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:23:29.234536 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230357    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230360    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230364    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230367    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230371    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230373    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230376    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230379    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230381    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230384    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230387    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230389    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230392    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230394    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230397    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230400    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230404    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230408    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230410    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:23:29.235040 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230413    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230416    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230418    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230420    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230423    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230425    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230428    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230430    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230433    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230435    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230438    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230440    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230443    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230445    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230447    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230450    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.230452    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230531    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230540    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230547    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230551    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 10:23:29.235511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230555    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230558    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230563    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230568    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230571    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230574    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230578    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230582    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230585    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230587    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230590    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230593    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230596    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230599    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230607    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230609    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230612    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230615    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230619    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230623    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230625    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230628    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230632    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230635    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 10:23:29.236016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230637    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230640    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230643    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230646    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230650    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230653    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230656    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230658    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230661    2578 flags.go:64] FLAG: --enable-server="true"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230664    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230669    2578 flags.go:64] FLAG: --event-burst="100"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230672    2578 flags.go:64] FLAG: --event-qps="50"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230675    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230678    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230681    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230685    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230688    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230691    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230694    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230697    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230700    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230704    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230710    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230715    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230719    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 10:23:29.236627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230724    2578 flags.go:64] FLAG: --feature-gates=""
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230730    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230734    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230740    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230743    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230746    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230749    2578 flags.go:64] FLAG: --help="false"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230753    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230756    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230759    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230761    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230765    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230768    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230771    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230773    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230776    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230780    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230783    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230786    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230789    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230792    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230795    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230798    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230801    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 10:23:29.237292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230803    2578 flags.go:64] FLAG: --lock-file=""
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230806    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230809    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230812    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230818    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230823    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230826    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230828    2578 flags.go:64] FLAG: --logging-format="text"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230831    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230834    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230837    2578 flags.go:64] FLAG: --manifest-url=""
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230840    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230844    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230847    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230851    2578 flags.go:64] FLAG: --max-pods="110"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230854    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230857    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230860    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230863    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230866    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230869    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230871    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230880    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230883    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230888    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 10:23:29.237871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230891    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230894    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230900    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230902    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230905    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230908    2578 flags.go:64] FLAG: --port="10250"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230911    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230914    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04385a220ec3c4cec"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230917    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230920    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230923    2578 flags.go:64] FLAG: --register-node="true"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230926    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230929    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230934    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230936    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230939    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230943    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230946    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230949    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230952    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230954    2578 flags.go:64] FLAG: --runonce="false"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230957    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230960    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230963    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230966    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 10:23:29.238579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230968    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230971    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230974    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230978    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230980    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230983    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230987    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230990    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230993    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230996    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.230998    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231004    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231007    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231009    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231014    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231016    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231019    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231022    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231024    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231027    2578 flags.go:64] FLAG: --v="2"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231038    2578 flags.go:64] FLAG: --version="false"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231043    2578 flags.go:64] FLAG: --vmodule=""
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231047    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.231050    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231173    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:23:29.239178 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231177    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231180    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231183    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231186    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231189    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231191    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231194    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231196    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231199    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231201    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231204    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231207    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231209    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231213    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231216    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231218    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231220    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231223    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231225    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:23:29.239870 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231228    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231230    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231232    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231235    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231237    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231240    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231242    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231245    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231249    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231251    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231254    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231257    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231259    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231262    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231264    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231280    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231283    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231286    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231290    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:23:29.240344 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231293    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231296    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231299    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231301    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231304    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231306    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231309    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231313    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231316    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231319    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231321    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231324    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231326    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231329    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231331    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231334    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231336    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231339    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231341    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231344    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:23:29.240825 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231347    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231351    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231353    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231356    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231359    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231362    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231364    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231367    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231369    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231372    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231374    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231377    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231379    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231382    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231384    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231387    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231389    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231391    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231394    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231400    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:23:29.241330 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231403    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231406    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231409    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231411    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231414    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231417    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231420    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231422    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231425    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231429    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231432    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231435    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231438    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231442    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231445    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231447    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231450    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231453    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231456    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:23:29.241847 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.231458    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.232114    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.238850    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.238874    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.238896    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.238902    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.238960    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.238965    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.238968    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.238972    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.238995    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239000    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239004    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239007    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:23:29.242321 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239010    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239013    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239016    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239019    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239021    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239024    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239026    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239030    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239034    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239037    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239040    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239042    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239045    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239048    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239051    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239054    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239058    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239060    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239063    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:23:29.242666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239068    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239071    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239074    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239077    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239079    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239083    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239085    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239088    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239090    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239093    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239095    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239098    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239101    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239104    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239106    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239109    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239111    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239114    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239116    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239119    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:23:29.243140 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239122    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239124    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239127    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239129    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239132    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239134    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239137    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239139    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239142    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239145    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239148    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239151    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239153    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239156    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239158    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239161    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239164    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239166    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239169    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239171    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:23:29.243633 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239174    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239177    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239179    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239182    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239185    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239187    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239190    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239194    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239198    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239201    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239203    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239206    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239209    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239211    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239214    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239217    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239220    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239223    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239225    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239228    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:23:29.244114 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239231    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239233    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239236    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239606    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239611    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239614    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239616    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239619    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239622    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239624    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239627    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239629    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.239635    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239749    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:23:29.244636 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239753    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239756    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239759    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239762    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239764    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239767    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239771    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239774    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239777    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239780    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239783    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239785    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239788    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239791    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239793    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239796    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239798    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239802    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239804    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239807    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:23:29.245018 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239810    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239812    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239815    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239818    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239820    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239822    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239825    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239827    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239830    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239832    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239835    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239838    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239841    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239844    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239847    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239849    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239851    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239854    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239856    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:23:29.245518 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239859    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239861    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239865    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239868    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239871    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239873    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239876    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239879    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239882    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239885    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239889    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239892    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239894    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239896    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239899    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239901    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239904    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239907    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239909    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:23:29.245970 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239911    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239914    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239916    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239919    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239921    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239923    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239926    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239928    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239930    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239933    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239935    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239938    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239940    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239942    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239945    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239947    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239950    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239952    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239955    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239957    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:23:29.246548 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239959    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239962    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239964    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239967    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239969    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239972    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239974    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239977    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239979    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239981    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239984    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239987    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239990    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239992    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239994    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239997    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.239999    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.240001    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.240004    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:23:29.247024 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:29.240006    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.240011    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.240133    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.242113    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.243688    2578 server.go:1019] "Starting client certificate rotation"
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.243788    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.243826    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.246241    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:23:29.247541 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.247403    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:23:29.261575 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.261533    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 28 10:23:29.263544 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.263528    2578 log.go:25] "Validated CRI v1 image API"
Jul 28 10:23:29.264375 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.264360    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 10:23:29.269006 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.268987    2578 fs.go:135] Filesystem UUIDs: map[279f9cea-8256-4f18-86dc-78073f38cb6b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cebc20e5-86c2-48cf-8847-76fa97a423a9:/dev/nvme0n1p4]
Jul 28 10:23:29.269062 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.269006    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 10:23:29.274455 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.274340    2578 manager.go:216] Machine: {Timestamp:2026-07-28 10:23:29.273086686 +0000 UTC m=+0.315178829 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100717 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29c939a8d9ede9b3d6f75f524b5ddd SystemUUID:ec29c939-a8d9-ede9-b3d6-f75f524b5ddd BootID:864d9620-df8a-4df7-b11f-035911f53cb5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:9d:06:31:67:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:9d:06:31:67:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:62:d7:f5:9d:45:14 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 10:23:29.274455 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.274448    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 10:23:29.274578 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.274529    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 10:23:29.275561 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.275533    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 10:23:29.275634 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.275546    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:23:29.275733 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.275564    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-67.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 10:23:29.276367 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.276357    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 10:23:29.276408 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.276370    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 10:23:29.276408 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.276377    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:23:29.277139 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.277127    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:23:29.278374 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.278363    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 10:23:29.282406 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282389    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:23:29.282550 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282539    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 10:23:29.282582 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282574    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 10:23:29.282613 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282587    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 10:23:29.282613 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282599    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 10:23:29.282613 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282608    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 28 10:23:29.282692 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.282625    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 10:23:29.283024 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.283012    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:23:29.283066 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.283030    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:23:29.285133 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.285113    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 10:23:29.286433 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.286416    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 10:23:29.286531 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.286455    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 10:23:29.287081 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287067    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 10:23:29.287141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287087    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 10:23:29.287141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287096    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 10:23:29.287141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287104    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 10:23:29.287141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287117    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 10:23:29.287141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287125    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 10:23:29.287141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287138    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 10:23:29.287367 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287148    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 10:23:29.287367 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287165    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 10:23:29.287367 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287174    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 10:23:29.287367 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287188    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 10:23:29.287367 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.287200    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 10:23:29.288450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.288433    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 10:23:29.288450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.288451    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 10:23:29.292340 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.292105    2578 server.go:1268] "Started kubelet"
Jul 28 10:23:29.292340 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.292107    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-67.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 10:23:29.292340 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.292216    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 10:23:29.292550 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.292311    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 10:23:29.295738 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.292303    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 10:23:29.296336 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.296306    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 10:23:29.296436 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.296358    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 10:23:29.296696 ip-10-0-131-67 systemd[1]: Started Kubernetes Kubelet.
Jul 28 10:23:29.300080 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.299696    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 10:23:29.301606 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.300309    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-67.ec2.internal.18c66d0e05bac934  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-67.ec2.internal,UID:ip-10-0-131-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-67.ec2.internal,},FirstTimestamp:2026-07-28 10:23:29.292077364 +0000 UTC m=+0.334169508,LastTimestamp:2026-07-28 10:23:29.292077364 +0000 UTC m=+0.334169508,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-67.ec2.internal,}"
Jul 28 10:23:29.301732 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.301692    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 10:23:29.302409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.302385    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwvjg"
Jul 28 10:23:29.303083 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.303061    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 10:23:29.307415 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.307398    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 10:23:29.307482 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.307413    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 10:23:29.307532 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.307507    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.307998 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.307981    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 10:23:29.307998 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.307999    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 10:23:29.308168 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308104    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 10:23:29.308168 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308113    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 10:23:29.308363 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308350    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 10:23:29.308565 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308550    2578 factory.go:55] Registering systemd factory
Jul 28 10:23:29.308611 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308570    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 28 10:23:29.308784 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308771    2578 factory.go:153] Registering CRI-O factory
Jul 28 10:23:29.308832 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308787    2578 factory.go:223] Registration of the crio container factory successfully
Jul 28 10:23:29.308864 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308837    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 10:23:29.308893 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308862    2578 factory.go:103] Registering Raw factory
Jul 28 10:23:29.308922 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308894    2578 manager.go:1196] Started watching for new ooms in manager
Jul 28 10:23:29.308973 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.308958    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qwvjg"
Jul 28 10:23:29.309465 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.309450    2578 manager.go:349] Starting recovery of all containers
Jul 28 10:23:29.315579 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.315551    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 10:23:29.315708 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.315662    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-67.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 10:23:29.319830 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.319607    2578 manager.go:354] Recovery completed
Jul 28 10:23:29.324556 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.324543    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.326431 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.326414    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.326510 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.326446    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.326510 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.326461    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.326985 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.326971    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 10:23:29.326985 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.326984    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 10:23:29.327077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.327001    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:23:29.327077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.327005    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.327777 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.327756    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.327816 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.327786    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.327816 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.327800    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.328718 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.328643    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-67.ec2.internal.18c66d0e07c6f995  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-67.ec2.internal,UID:ip-10-0-131-67.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-67.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-67.ec2.internal,},FirstTimestamp:2026-07-28 10:23:29.326430613 +0000 UTC m=+0.368522757,LastTimestamp:2026-07-28 10:23:29.326430613 +0000 UTC m=+0.368522757,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-67.ec2.internal,}"
Jul 28 10:23:29.329160 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.329149    2578 policy_none.go:49] "None policy: Start"
Jul 28 10:23:29.329214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.329164    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 10:23:29.329214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.329175    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 10:23:29.331279 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.331252    2578 policy_none.go:47] "Start"
Jul 28 10:23:29.378006 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.377985    2578 manager.go:337] "Starting Device Plugin manager"
Jul 28 10:23:29.378151 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.378028    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 10:23:29.378151 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.378038    2578 server.go:87] "Starting device plugin registration server"
Jul 28 10:23:29.378377 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.378363    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 10:23:29.378422 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.378380    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 10:23:29.378491 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.378471    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 10:23:29.378616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.378587    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 10:23:29.378616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.378613    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 10:23:29.379112 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.379097    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 10:23:29.379167 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.379137    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.478798 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.478708    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.480846 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.479751    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.480846 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.479790    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.480846 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.479804    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.480846 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.479836    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.485955 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.485934    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.486057 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.485962    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-67.ec2.internal\": node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.486537 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.486509    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 10:23:29.487921 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.487903    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 10:23:29.488007 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.487943    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 10:23:29.488007 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.487965    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 10:23:29.488109 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.488005    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 10:23:29.489802 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.489782    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:29.502057 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.502032    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.588614 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.588558    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"]
Jul 28 10:23:29.588714 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.588663    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.590462 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.590446    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.590532 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.590477    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.590532 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.590487    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.592665 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.592649    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.592782 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.592766    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.592840 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.592803    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.593482 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.593467    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.593552 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.593498    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.593552 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.593508    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.593552 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.593471    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.593639 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.593571    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.593639 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.593586    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.595648 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.595631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.595751 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.595660    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:23:29.596446 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.596431    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:23:29.596536 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.596458    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:23:29.596536 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.596471    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:23:29.602856 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.602838    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.609754 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.609728    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3fc6426ab408b4eb2e063772fac6b7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"e3fc6426ab408b4eb2e063772fac6b7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.609837 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.609757    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3fc6426ab408b4eb2e063772fac6b7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"e3fc6426ab408b4eb2e063772fac6b7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.609837 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.609775    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b28dbc1bfe8498d00f22a31b5ebf594-config\") pod \"kube-apiserver-proxy-ip-10-0-131-67.ec2.internal\" (UID: \"1b28dbc1bfe8498d00f22a31b5ebf594\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.614015 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.613997    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-67.ec2.internal\" not found" node="ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.617641 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.617622    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-67.ec2.internal\" not found" node="ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.703365 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.703331    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.710682 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.710656    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b28dbc1bfe8498d00f22a31b5ebf594-config\") pod \"kube-apiserver-proxy-ip-10-0-131-67.ec2.internal\" (UID: \"1b28dbc1bfe8498d00f22a31b5ebf594\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.710760 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.710686    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3fc6426ab408b4eb2e063772fac6b7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"e3fc6426ab408b4eb2e063772fac6b7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.710760 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.710704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3fc6426ab408b4eb2e063772fac6b7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"e3fc6426ab408b4eb2e063772fac6b7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.710831 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.710759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3fc6426ab408b4eb2e063772fac6b7e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"e3fc6426ab408b4eb2e063772fac6b7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.710831 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.710759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b28dbc1bfe8498d00f22a31b5ebf594-config\") pod \"kube-apiserver-proxy-ip-10-0-131-67.ec2.internal\" (UID: \"1b28dbc1bfe8498d00f22a31b5ebf594\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.710831 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.710797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e3fc6426ab408b4eb2e063772fac6b7e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal\" (UID: \"e3fc6426ab408b4eb2e063772fac6b7e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.803480 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.803399    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.908615 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:29.908563    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:29.916390 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.916368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:29.920212 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:29.920189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:30.010730 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.010685    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-67.ec2.internal\" not found"
Jul 28 10:23:30.108163 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.108065    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:30.158203 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.158169    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:30.208240 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.208206    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:30.219880 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.219848    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:23:30.221406 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.221389    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal"
Jul 28 10:23:30.239775 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.239751    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:23:30.243111 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.243089    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 10:23:30.243240 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.243219    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:23:30.243279 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.243220    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:23:30.243279 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.243225    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:23:30.285593 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.285560    2578 apiserver.go:52] "Watching apiserver"
Jul 28 10:23:30.301591 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.301553    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 10:23:30.301965 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.301939    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-q5h4z","openshift-image-registry/node-ca-k44hl","openshift-multus/multus-cvrqj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal","openshift-multus/multus-additional-cni-plugins-dstdj","openshift-multus/network-metrics-daemon-d5g2m","openshift-network-diagnostics/network-check-target-nsc54","openshift-network-operator/iptables-alerter-cg6x9","openshift-ovn-kubernetes/ovnkube-node-jmrkn","kube-system/global-pull-secret-syncer-d4s8q","kube-system/konnectivity-agent-h55h6","kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm","openshift-cluster-node-tuning-operator/tuned-5p5bx"]
Jul 28 10:23:30.304601 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.304579    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:30.304719 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.304650    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:30.306852 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.306835    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.309210 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.307548    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 10:23:30.309946 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.309912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-f5hhx\""
Jul 28 10:23:30.310679 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.310657    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.311645 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.311622    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 10:23:30.311736 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.311633    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 10:23:30.312893 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.312849    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.312893 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.312888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.313050 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.312982    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.313470 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.313446    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 10:23:30.313704 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.313684    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 10:23:30.313805 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.313721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wmkp9\""
Jul 28 10:23:30.315192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-cni-multus\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-etc-kubernetes\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.315286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315245    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7016e81-0f35-49e8-9319-c906df347a4f-konnectivity-ca\") pod \"konnectivity-agent-h55h6\" (UID: \"c7016e81-0f35-49e8-9319-c906df347a4f\") " pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.315286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315297    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-cni-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315324    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-k8s-cni-cncf-io\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-netns\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315400    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-os-release\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315426    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c92fc1-af01-49a6-b40b-c6fddc21c333-cni-binary-copy\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315451    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-kubelet\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315466    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-multus-certs\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315505    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wxv8\" (UniqueName: \"kubernetes.io/projected/11c92fc1-af01-49a6-b40b-c6fddc21c333-kube-api-access-2wxv8\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315546    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-system-cni-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-hostroot\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-conf-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-daemon-config\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315629    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315646    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7016e81-0f35-49e8-9319-c906df347a4f-agent-certs\") pod \"konnectivity-agent-h55h6\" (UID: \"c7016e81-0f35-49e8-9319-c906df347a4f\") " pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315668    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-cnibin\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-socket-dir-parent\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.315841 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.315731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-cni-bin\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.316153 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.316093    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.316153 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.316094    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 10:23:30.316806 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.316789    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-d5vlj\""
Jul 28 10:23:30.317570 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.317540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:30.317666 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.317615    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:30.317819 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.317801    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 10:23:30.317965 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.317947    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rwtdh\""
Jul 28 10:23:30.318035 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.317999    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 10:23:30.318221 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.318204    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:23:30.320772 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.320732    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:18:29 +0000 UTC" deadline="2028-02-14 11:22:46.42430134 +0000 UTC"
Jul 28 10:23:30.320851 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.320773    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13584h59m16.103531776s"
Jul 28 10:23:30.320851 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.320846    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.323380 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.323354    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.323482 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.323423    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.323482 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.323429    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-s47n8\""
Jul 28 10:23:30.323610 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.323592    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.325993 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.325972    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.326083 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.326041    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:30.326401 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.326384    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 10:23:30.326479 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.326460    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.326525 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.326505    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 10:23:30.327789 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.327770    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qnts6\""
Jul 28 10:23:30.327885 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.327861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 10:23:30.327885 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.327876    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.328040 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.327893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 10:23:30.329514 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.329481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.331769 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.331751    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.331853 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.331815    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 10:23:30.332470 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.332452    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.332724 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.332701    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.332807 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.332702    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lsrk6\""
Jul 28 10:23:30.334874 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.334856    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9j7gw\""
Jul 28 10:23:30.335051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.335035    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.335621 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.335604    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.335731 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.335709    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.335823 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.335800    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 10:23:30.339627 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.339611    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hp846\""
Jul 28 10:23:30.339809 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.339793    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:23:30.339873 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.339864    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 10:23:30.347161 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.346871    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6mfg"
Jul 28 10:23:30.358082 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.358060    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m6mfg"
Jul 28 10:23:30.410227 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.410194    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-system-cni-dir\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416439    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87jw5\" (UniqueName: \"kubernetes.io/projected/5d9439f1-1ac5-466a-bace-3b399fed2355-kube-api-access-87jw5\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovn-node-metrics-cert\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysctl-conf\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416519    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-systemd\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e94f9f5-929b-4d0d-9162-716538c46090-etc-tuned\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wxv8\" (UniqueName: \"kubernetes.io/projected/11c92fc1-af01-49a6-b40b-c6fddc21c333-kube-api-access-2wxv8\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416639    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-cni-bin\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416661    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8jmg\" (UniqueName: \"kubernetes.io/projected/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-kube-api-access-q8jmg\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.416688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416713    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d9439f1-1ac5-466a-bace-3b399fed2355-host-slash\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416743    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovnkube-config\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-serviceca\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416792    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7016e81-0f35-49e8-9319-c906df347a4f-agent-certs\") pod \"konnectivity-agent-h55h6\" (UID: \"c7016e81-0f35-49e8-9319-c906df347a4f\") " pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-log-socket\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416841    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6k6l\" (UniqueName: \"kubernetes.io/projected/24a56284-b1d8-41d0-8f05-a14d507b6e4a-kube-api-access-p6k6l\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-systemd\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416906    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-var-lib-kubelet\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416938    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a56284-b1d8-41d0-8f05-a14d507b6e4a-hosts-file\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/081537ec-b941-46e9-b73e-f156ddb7585c-kubelet-config\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.416993    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-conf-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-kubelet\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-etc-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cnibin\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.417319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417295    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-conf-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417340    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-cnibin\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417381    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-cni-multus\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-cnibin\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417388    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-cni-multus\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417509    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d9439f1-1ac5-466a-bace-3b399fed2355-iptables-alerter-script\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417550    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-run\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e94f9f5-929b-4d0d-9162-716538c46090-tmp\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvrx\" (UniqueName: \"kubernetes.io/projected/04b03a5e-2ba6-4e19-80f0-e24aea151a63-kube-api-access-jcvrx\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417722    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7016e81-0f35-49e8-9319-c906df347a4f-konnectivity-ca\") pod \"konnectivity-agent-h55h6\" (UID: \"c7016e81-0f35-49e8-9319-c906df347a4f\") " pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-ovn\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417793    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-env-overrides\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417821    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-os-release\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-kubelet\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.418142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-multus-certs\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417954    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-multus-certs\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-node-log\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.417989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-kubelet\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-os-release\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418060    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-cni-netd\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418086    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-modprobe-d\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cni-binary-copy\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-system-cni-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-hostroot\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418201    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-system-cni-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-systemd-units\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418235    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-hostroot\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovnkube-script-lib\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418275    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkmk7\" (UniqueName: \"kubernetes.io/projected/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-kube-api-access-pkmk7\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418302    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-socket-dir-parent\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-cni-bin\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-host\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418409    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-socket-dir-parent\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418411    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c7016e81-0f35-49e8-9319-c906df347a4f-konnectivity-ca\") pod \"konnectivity-agent-h55h6\" (UID: \"c7016e81-0f35-49e8-9319-c906df347a4f\") " pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b68dg\" (UniqueName: \"kubernetes.io/projected/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-kube-api-access-b68dg\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-os-release\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-var-lib-cni-bin\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418506    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-cni-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-k8s-cni-cncf-io\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418555    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-netns\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418587    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a56284-b1d8-41d0-8f05-a14d507b6e4a-tmp-dir\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-k8s-cni-cncf-io\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418594    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-cni-dir\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/081537ec-b941-46e9-b73e-f156ddb7585c-dbus\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-host-run-netns\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-run-netns\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.419978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-device-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c92fc1-af01-49a6-b40b-c6fddc21c333-cni-binary-copy\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-registration-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418829    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysconfig\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-kubernetes\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418900    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysctl-d\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-host\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzrf\" (UniqueName: \"kubernetes.io/projected/0e94f9f5-929b-4d0d-9162-716538c46090-kube-api-access-dmzrf\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.418976    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-daemon-config\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419001    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z897p\" (UniqueName: \"kubernetes.io/projected/c9311b88-8997-40d9-a0ea-5f5b5beca84f-kube-api-access-z897p\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419028    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-var-lib-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419053    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419108    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-etc-kubernetes\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419132    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-slash\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.420919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-socket-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.421637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-sys-fs\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.421637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419204    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-sys\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.421637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419222    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/11c92fc1-af01-49a6-b40b-c6fddc21c333-cni-binary-copy\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.421637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419227    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-lib-modules\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.421637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419356    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/11c92fc1-af01-49a6-b40b-c6fddc21c333-etc-kubernetes\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.421637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.419527    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/11c92fc1-af01-49a6-b40b-c6fddc21c333-multus-daemon-config\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.421957 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.421938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c7016e81-0f35-49e8-9319-c906df347a4f-agent-certs\") pod \"konnectivity-agent-h55h6\" (UID: \"c7016e81-0f35-49e8-9319-c906df347a4f\") " pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.436084 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.436053    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:23:30.436241 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.436091    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:23:30.436241 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.436107    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:30.436241 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.436196    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:23:30.936168819 +0000 UTC m=+1.978260952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:30.436954 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.436927    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:30.443802 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.443765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wxv8\" (UniqueName: \"kubernetes.io/projected/11c92fc1-af01-49a6-b40b-c6fddc21c333-kube-api-access-2wxv8\") pod \"multus-cvrqj\" (UID: \"11c92fc1-af01-49a6-b40b-c6fddc21c333\") " pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.504104 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.504061    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3fc6426ab408b4eb2e063772fac6b7e.slice/crio-441cb89b7ad2b3c175ee82fb85ef5dd126a6efd64992897e3d2f1ae2e5558661 WatchSource:0}: Error finding container 441cb89b7ad2b3c175ee82fb85ef5dd126a6efd64992897e3d2f1ae2e5558661: Status 404 returned error can't find the container with id 441cb89b7ad2b3c175ee82fb85ef5dd126a6efd64992897e3d2f1ae2e5558661
Jul 28 10:23:30.504392 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.504369    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b28dbc1bfe8498d00f22a31b5ebf594.slice/crio-c4e950a6856ceaa23b993c2c9d3237842569da8067a67ac07c9b02dca2be795e WatchSource:0}: Error finding container c4e950a6856ceaa23b993c2c9d3237842569da8067a67ac07c9b02dca2be795e: Status 404 returned error can't find the container with id c4e950a6856ceaa23b993c2c9d3237842569da8067a67ac07c9b02dca2be795e
Jul 28 10:23:30.509150 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.509132    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:23:30.520228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkmk7\" (UniqueName: \"kubernetes.io/projected/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-kube-api-access-pkmk7\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.520228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520225    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-host\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.520492 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b68dg\" (UniqueName: \"kubernetes.io/projected/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-kube-api-access-b68dg\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.520492 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-host\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.520492 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-os-release\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.520660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.520660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.520660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520599    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-os-release\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.520660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520642    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a56284-b1d8-41d0-8f05-a14d507b6e4a-tmp-dir\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/081537ec-b941-46e9-b73e-f156ddb7585c-dbus\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-run-netns\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-device-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-run-netns\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520794    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-device-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520849    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-registration-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.520871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysconfig\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-kubernetes\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520872    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/081537ec-b941-46e9-b73e-f156ddb7585c-dbus\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.520953    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-kubernetes\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520961    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysconfig\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520983    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/24a56284-b1d8-41d0-8f05-a14d507b6e4a-tmp-dir\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.520973    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-registration-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysctl-d\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.521027    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:31.021008828 +0000 UTC m=+2.063100978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-host\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzrf\" (UniqueName: \"kubernetes.io/projected/0e94f9f5-929b-4d0d-9162-716538c46090-kube-api-access-dmzrf\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z897p\" (UniqueName: \"kubernetes.io/projected/c9311b88-8997-40d9-a0ea-5f5b5beca84f-kube-api-access-z897p\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521115    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysctl-d\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-var-lib-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521136    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-host\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.521260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-var-lib-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-slash\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521238    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-slash\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-run-ovn-kubernetes\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-socket-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-sys-fs\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521404    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-sys\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-lib-modules\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-system-cni-dir\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521456    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-sys-fs\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jw5\" (UniqueName: \"kubernetes.io/projected/5d9439f1-1ac5-466a-bace-3b399fed2355-kube-api-access-87jw5\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-socket-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-system-cni-dir\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-sys\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovn-node-metrics-cert\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521568    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysctl-conf\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521586    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-systemd\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-lib-modules\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e94f9f5-929b-4d0d-9162-716538c46090-etc-tuned\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521632    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-systemd\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-cni-bin\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521676    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8jmg\" (UniqueName: \"kubernetes.io/projected/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-kube-api-access-q8jmg\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521699    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521726    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d9439f1-1ac5-466a-bace-3b399fed2355-host-slash\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521736    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-cni-bin\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-sysctl-conf\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovnkube-config\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521799    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-serviceca\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521807    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5d9439f1-1ac5-466a-bace-3b399fed2355-host-slash\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-log-socket\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521862    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6k6l\" (UniqueName: \"kubernetes.io/projected/24a56284-b1d8-41d0-8f05-a14d507b6e4a-kube-api-access-p6k6l\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521887    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-systemd\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-var-lib-kubelet\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.522980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521913    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.521934    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a56284-b1d8-41d0-8f05-a14d507b6e4a-hosts-file\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/081537ec-b941-46e9-b73e-f156ddb7585c-kubelet-config\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522039    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-kubelet\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-etc-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-log-socket\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cnibin\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cnibin\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d9439f1-1ac5-466a-bace-3b399fed2355-iptables-alerter-script\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-systemd\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522266    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-run\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e94f9f5-929b-4d0d-9162-716538c46090-tmp\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522305    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-var-lib-kubelet\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522375    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/24a56284-b1d8-41d0-8f05-a14d507b6e4a-hosts-file\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.523801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522412    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/081537ec-b941-46e9-b73e-f156ddb7585c-kubelet-config\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522558    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522552    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-run\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522475    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovnkube-config\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522560    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvrx\" (UniqueName: \"kubernetes.io/projected/04b03a5e-2ba6-4e19-80f0-e24aea151a63-kube-api-access-jcvrx\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522625    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-ovn\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-env-overrides\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522714    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-serviceca\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-node-log\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-run-ovn\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522785    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-node-log\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522823    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-cni-netd\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522853    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-modprobe-d\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522879    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cni-binary-copy\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.524691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522937    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-systemd-units\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.522963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovnkube-script-lib\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5d9439f1-1ac5-466a-bace-3b399fed2355-iptables-alerter-script\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523244    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-env-overrides\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-etc-openvswitch\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523398    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0e94f9f5-929b-4d0d-9162-716538c46090-etc-modprobe-d\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523451    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523497    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-cni-netd\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-host-kubelet\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523571    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovnkube-script-lib\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.523603    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.523632    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-systemd-units\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:30.523670    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:23:31.023651166 +0000 UTC m=+2.065743301 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.524029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04b03a5e-2ba6-4e19-80f0-e24aea151a63-cni-binary-copy\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.524618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-ovn-node-metrics-cert\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.525303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.524626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0e94f9f5-929b-4d0d-9162-716538c46090-etc-tuned\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.525979 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.525934    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e94f9f5-929b-4d0d-9162-716538c46090-tmp\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.530757 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.530723    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkmk7\" (UniqueName: \"kubernetes.io/projected/613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f-kube-api-access-pkmk7\") pod \"aws-ebs-csi-driver-node-fcfhm\" (UID: \"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.530986 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.530966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z897p\" (UniqueName: \"kubernetes.io/projected/c9311b88-8997-40d9-a0ea-5f5b5beca84f-kube-api-access-z897p\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:30.531465 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.531443    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzrf\" (UniqueName: \"kubernetes.io/projected/0e94f9f5-929b-4d0d-9162-716538c46090-kube-api-access-dmzrf\") pod \"tuned-5p5bx\" (UID: \"0e94f9f5-929b-4d0d-9162-716538c46090\") " pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.532088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.532071    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b68dg\" (UniqueName: \"kubernetes.io/projected/9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd-kube-api-access-b68dg\") pod \"node-ca-k44hl\" (UID: \"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd\") " pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.537480 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.537445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6k6l\" (UniqueName: \"kubernetes.io/projected/24a56284-b1d8-41d0-8f05-a14d507b6e4a-kube-api-access-p6k6l\") pod \"node-resolver-q5h4z\" (UID: \"24a56284-b1d8-41d0-8f05-a14d507b6e4a\") " pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.537789 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.537767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87jw5\" (UniqueName: \"kubernetes.io/projected/5d9439f1-1ac5-466a-bace-3b399fed2355-kube-api-access-87jw5\") pod \"iptables-alerter-cg6x9\" (UID: \"5d9439f1-1ac5-466a-bace-3b399fed2355\") " pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.537981 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.537963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8jmg\" (UniqueName: \"kubernetes.io/projected/e3c17446-0ae7-4e7f-bbf6-669b2625cbed-kube-api-access-q8jmg\") pod \"ovnkube-node-jmrkn\" (UID: \"e3c17446-0ae7-4e7f-bbf6-669b2625cbed\") " pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.538038 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.538022    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvrx\" (UniqueName: \"kubernetes.io/projected/04b03a5e-2ba6-4e19-80f0-e24aea151a63-kube-api-access-jcvrx\") pod \"multus-additional-cni-plugins-dstdj\" (UID: \"04b03a5e-2ba6-4e19-80f0-e24aea151a63\") " pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.638563 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.638457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:30.649932 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.649904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7016e81_0f35_49e8_9319_c906df347a4f.slice/crio-cef237d8f34b4d3fb10f22edc635154408b1d3dd0d95234506862d755e5047eb WatchSource:0}: Error finding container cef237d8f34b4d3fb10f22edc635154408b1d3dd0d95234506862d755e5047eb: Status 404 returned error can't find the container with id cef237d8f34b4d3fb10f22edc635154408b1d3dd0d95234506862d755e5047eb
Jul 28 10:23:30.653370 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.653347    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cvrqj"
Jul 28 10:23:30.662718 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.662688    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c92fc1_af01_49a6_b40b_c6fddc21c333.slice/crio-433026de24b97aa82fdea05d252c33c2c25db966217d78b18947440f6f033fda WatchSource:0}: Error finding container 433026de24b97aa82fdea05d252c33c2c25db966217d78b18947440f6f033fda: Status 404 returned error can't find the container with id 433026de24b97aa82fdea05d252c33c2c25db966217d78b18947440f6f033fda
Jul 28 10:23:30.663244 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.663210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cg6x9"
Jul 28 10:23:30.673698 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.673661    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9439f1_1ac5_466a_bace_3b399fed2355.slice/crio-36abcb1134d1326cd993662bd81f2c373dec8e0d6a58b8893892d3524ae87bd2 WatchSource:0}: Error finding container 36abcb1134d1326cd993662bd81f2c373dec8e0d6a58b8893892d3524ae87bd2: Status 404 returned error can't find the container with id 36abcb1134d1326cd993662bd81f2c373dec8e0d6a58b8893892d3524ae87bd2
Jul 28 10:23:30.685486 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.685458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dstdj"
Jul 28 10:23:30.695454 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.695419    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b03a5e_2ba6_4e19_80f0_e24aea151a63.slice/crio-ac0c3212063cde1717755a1413a278592dca4a4df597803f23258b122d898636 WatchSource:0}: Error finding container ac0c3212063cde1717755a1413a278592dca4a4df597803f23258b122d898636: Status 404 returned error can't find the container with id ac0c3212063cde1717755a1413a278592dca4a4df597803f23258b122d898636
Jul 28 10:23:30.699960 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.699935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q5h4z"
Jul 28 10:23:30.709902 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.709873    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a56284_b1d8_41d0_8f05_a14d507b6e4a.slice/crio-52a7b9732a536d11cf0fbd4b8e62e2cf0fb10c633c4061a1aced95480a47ace7 WatchSource:0}: Error finding container 52a7b9732a536d11cf0fbd4b8e62e2cf0fb10c633c4061a1aced95480a47ace7: Status 404 returned error can't find the container with id 52a7b9732a536d11cf0fbd4b8e62e2cf0fb10c633c4061a1aced95480a47ace7
Jul 28 10:23:30.714689 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.714663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:30.724566 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.724534    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c17446_0ae7_4e7f_bbf6_669b2625cbed.slice/crio-cc1de27798a534e9002298309d0e5f3446be3073603b49473baebf45dde35bdf WatchSource:0}: Error finding container cc1de27798a534e9002298309d0e5f3446be3073603b49473baebf45dde35bdf: Status 404 returned error can't find the container with id cc1de27798a534e9002298309d0e5f3446be3073603b49473baebf45dde35bdf
Jul 28 10:23:30.734180 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.734138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k44hl"
Jul 28 10:23:30.743596 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.743563    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b4d85bf_16e0_4b73_a1fc_fb1863ef6cfd.slice/crio-84115a5562d5dd9339eb6b0b8891d1f42c2a44e09f4d38a3f17cf72f1c69e083 WatchSource:0}: Error finding container 84115a5562d5dd9339eb6b0b8891d1f42c2a44e09f4d38a3f17cf72f1c69e083: Status 404 returned error can't find the container with id 84115a5562d5dd9339eb6b0b8891d1f42c2a44e09f4d38a3f17cf72f1c69e083
Jul 28 10:23:30.750407 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.750384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm"
Jul 28 10:23:30.755723 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:30.755703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5p5bx"
Jul 28 10:23:30.762969 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.762944    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod613a23f3_d82b_4ba6_9dbe_2a6b2bd8709f.slice/crio-705265d15163034105ded1774bc14980dba0e8cfe4ec6a7e1ee52717b3a3891d WatchSource:0}: Error finding container 705265d15163034105ded1774bc14980dba0e8cfe4ec6a7e1ee52717b3a3891d: Status 404 returned error can't find the container with id 705265d15163034105ded1774bc14980dba0e8cfe4ec6a7e1ee52717b3a3891d
Jul 28 10:23:30.766079 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:23:30.766058    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e94f9f5_929b_4d0d_9162_716538c46090.slice/crio-a7927b8c031c05573ecb9e56cb2ce92c7b99125e1524818820bdca9c144e1fee WatchSource:0}: Error finding container a7927b8c031c05573ecb9e56cb2ce92c7b99125e1524818820bdca9c144e1fee: Status 404 returned error can't find the container with id a7927b8c031c05573ecb9e56cb2ce92c7b99125e1524818820bdca9c144e1fee
Jul 28 10:23:31.028145 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.028037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:31.028145 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.028101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:31.028145 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.028140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:31.028396 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028220    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:31.028396 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028291    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:23:31.028396 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028310    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:23:31.028396 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028325    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:31.028518 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028458    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:31.028518 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028310    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:23:32.028270386 +0000 UTC m=+3.070362535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:31.028590 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:23:32.028495765 +0000 UTC m=+3.070587902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:31.028590 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.028560    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:32.028548446 +0000 UTC m=+3.070640802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:31.362511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.362406    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:18:30 +0000 UTC" deadline="2028-02-12 06:32:24.620538145 +0000 UTC"
Jul 28 10:23:31.362511 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.362444    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13532h8m53.258099699s"
Jul 28 10:23:31.489637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.488994    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:31.489637 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.489155    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:31.489970 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.489940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:31.490134 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:31.490087    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:31.553776 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.553701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerStarted","Data":"ac0c3212063cde1717755a1413a278592dca4a4df597803f23258b122d898636"}
Jul 28 10:23:31.568659 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.568551    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h55h6" event={"ID":"c7016e81-0f35-49e8-9319-c906df347a4f","Type":"ContainerStarted","Data":"cef237d8f34b4d3fb10f22edc635154408b1d3dd0d95234506862d755e5047eb"}
Jul 28 10:23:31.599040 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.598996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" event={"ID":"1b28dbc1bfe8498d00f22a31b5ebf594","Type":"ContainerStarted","Data":"c4e950a6856ceaa23b993c2c9d3237842569da8067a67ac07c9b02dca2be795e"}
Jul 28 10:23:31.624230 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.624181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" event={"ID":"e3fc6426ab408b4eb2e063772fac6b7e","Type":"ContainerStarted","Data":"441cb89b7ad2b3c175ee82fb85ef5dd126a6efd64992897e3d2f1ae2e5558661"}
Jul 28 10:23:31.643044 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.643000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm" event={"ID":"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f","Type":"ContainerStarted","Data":"705265d15163034105ded1774bc14980dba0e8cfe4ec6a7e1ee52717b3a3891d"}
Jul 28 10:23:31.656309 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.656262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cg6x9" event={"ID":"5d9439f1-1ac5-466a-bace-3b399fed2355","Type":"ContainerStarted","Data":"36abcb1134d1326cd993662bd81f2c373dec8e0d6a58b8893892d3524ae87bd2"}
Jul 28 10:23:31.679380 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.679313    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cvrqj" event={"ID":"11c92fc1-af01-49a6-b40b-c6fddc21c333","Type":"ContainerStarted","Data":"433026de24b97aa82fdea05d252c33c2c25db966217d78b18947440f6f033fda"}
Jul 28 10:23:31.687900 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.687853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p5bx" event={"ID":"0e94f9f5-929b-4d0d-9162-716538c46090","Type":"ContainerStarted","Data":"a7927b8c031c05573ecb9e56cb2ce92c7b99125e1524818820bdca9c144e1fee"}
Jul 28 10:23:31.710815 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.710769    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k44hl" event={"ID":"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd","Type":"ContainerStarted","Data":"84115a5562d5dd9339eb6b0b8891d1f42c2a44e09f4d38a3f17cf72f1c69e083"}
Jul 28 10:23:31.737024 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.736893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"cc1de27798a534e9002298309d0e5f3446be3073603b49473baebf45dde35bdf"}
Jul 28 10:23:31.751053 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.751004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5h4z" event={"ID":"24a56284-b1d8-41d0-8f05-a14d507b6e4a","Type":"ContainerStarted","Data":"52a7b9732a536d11cf0fbd4b8e62e2cf0fb10c633c4061a1aced95480a47ace7"}
Jul 28 10:23:31.818522 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.818220    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:31.927337 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:31.926448    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.038684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.038750    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.038788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.038969    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.038992    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.039006    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.039071    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:23:34.039049233 +0000 UTC m=+5.081141370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.039149    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.039184    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:23:34.039171984 +0000 UTC m=+5.081264120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.039241    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:32.039717 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.039271    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:34.039260918 +0000 UTC m=+5.081353054 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:32.100501 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.100039    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:23:32.363404 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.363206    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:18:30 +0000 UTC" deadline="2028-04-11 02:09:14.28506385 +0000 UTC"
Jul 28 10:23:32.363404 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.363270    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14943h45m41.921798432s"
Jul 28 10:23:32.489270 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:32.489178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:32.489489 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:32.489409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:33.491704 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:33.491669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:33.492197 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:33.491815    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:33.492197 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:33.491903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:33.492197 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:33.491995    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:34.057024 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:34.056978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:34.057219 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:34.057081    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:34.057219 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:34.057124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:34.057384 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057276    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:34.057384 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057362    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:38.057341944 +0000 UTC m=+9.099434076 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:34.057515 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057463    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:23:34.057515 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057481    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:23:34.057515 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057495    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:34.057657 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057539    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:23:38.057524746 +0000 UTC m=+9.099616881 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:34.057657 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057602    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:34.057657 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.057635    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:23:38.057622314 +0000 UTC m=+9.099714450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:34.488581 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:34.488501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:34.488742 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:34.488645    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:35.489592 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:35.489563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:35.489988 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:35.489667    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:35.490081 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:35.490051    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:35.490191 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:35.490175    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:36.489250 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:36.489206    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:36.489466 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:36.489364    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:37.491002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:37.490699    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:37.491463 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:37.491037    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:37.491539 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:37.491499    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:37.491626 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:37.491605    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:38.091463 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:38.091425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:38.091650 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:38.091545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:38.091650 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091585    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:38.091650 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:38.091596    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091652    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:23:46.091633168 +0000 UTC m=+17.133725302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091679    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091707    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091724    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091736    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091774    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:23:46.09175483 +0000 UTC m=+17.133846960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:38.091796 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.091791    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:23:46.091783705 +0000 UTC m=+17.133875836 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:38.489292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:38.489192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:38.489447 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:38.489346    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:39.489115 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:39.489077    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:39.489564 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:39.489190    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:39.489564 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:39.489258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:39.489564 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:39.489358    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:40.488776 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:40.488735    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:40.488972 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:40.488873    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:41.489028 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:41.488991    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:41.489485 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:41.489108    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:41.489485 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:41.489170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:41.489485 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:41.489317    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:42.488565 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:42.488533    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:42.488734 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:42.488644    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:43.489252 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:43.489214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:43.489747 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:43.489262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:43.489747 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:43.489364    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:43.489747 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:43.489480    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:44.488759 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:44.488723    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:44.488942 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:44.488854    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:45.488644 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:45.488607    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:45.489091 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:45.488740    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:45.489091 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:45.488798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:45.489091 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:45.488947    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:46.154926 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:46.154891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:46.154926 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:46.154932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:46.154970    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155034    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155056    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155095    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:24:02.155079898 +0000 UTC m=+33.197172028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155109    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:24:02.155102115 +0000 UTC m=+33.197194247 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155105    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155134    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:23:46.155173 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155149    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:46.155586 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.155202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:24:02.155185052 +0000 UTC m=+33.197277208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:23:46.488631 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:46.488599    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:46.488871 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:46.488726    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:47.488257 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:47.488224    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:47.488445 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:47.488238    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:47.488445 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:47.488373    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:47.488524 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:47.488443    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:48.488766 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:48.488728    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:48.489180 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:48.488868    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:49.489201 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:49.489161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:49.489699 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:49.489245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:49.489699 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:49.489263    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:49.489699 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:49.489397    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:50.488220 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:50.488186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:50.488483 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:50.488313    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:51.489342 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.489316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:51.489817 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.489356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:51.489817 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:51.489461    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:51.490028 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:51.489882    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:51.824484 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.823181    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5p5bx" event={"ID":"0e94f9f5-929b-4d0d-9162-716538c46090","Type":"ContainerStarted","Data":"2678307976b9e711cd54f54f00ebd5023ea3e96280ccf13cec1533196df82bea"}
Jul 28 10:23:51.828175 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.828082    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:23:51.829132 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.828846    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3c17446-0ae7-4e7f-bbf6-669b2625cbed" containerID="3bd79cb3a4d064b73aa30d0cac81272876cbd17fffbe3cc783ba80be59e16763" exitCode=1
Jul 28 10:23:51.829132 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.828936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"43e928e5683f80838ff8ff299dfcabc930b8e52ec2b2a07c8d20948c3a9c2877"}
Jul 28 10:23:51.829132 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.829022    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerDied","Data":"3bd79cb3a4d064b73aa30d0cac81272876cbd17fffbe3cc783ba80be59e16763"}
Jul 28 10:23:51.829132 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.829074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"f791f294009f728e7cfcda6b4dfc60a5c5c854e4fefedf19b62de6c7a719856c"}
Jul 28 10:23:51.831352 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.831288    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" event={"ID":"1b28dbc1bfe8498d00f22a31b5ebf594","Type":"ContainerStarted","Data":"3c71179f85902e6f3f2542847af782d9acdf1efde8efd68f4b3b79146a6757fe"}
Jul 28 10:23:51.844443 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.844384    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5p5bx" podStartSLOduration=1.294074543 podStartE2EDuration="21.844364264s" totalImagesPullingTime="20.550289721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:51.844116477 +0000 UTC m=+22.886208633" watchObservedRunningTime="2026-07-28 10:23:51.844364264 +0000 UTC m=+22.886456417"
Jul 28 10:23:51.847609 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.847568    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cvrqj" event={"ID":"11c92fc1-af01-49a6-b40b-c6fddc21c333","Type":"ContainerStarted","Data":"2e21c210ac6bcecf843ab2fb83c7fd4611b2ea3ae44baa3a3c753d0c5ef92bf3"}
Jul 28 10:23:51.859228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:51.859170    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-67.ec2.internal" podStartSLOduration=21.859147737 podStartE2EDuration="21.859147737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:51.858152359 +0000 UTC m=+22.900244511" watchObservedRunningTime="2026-07-28 10:23:51.859147737 +0000 UTC m=+22.901239888"
Jul 28 10:23:52.488415 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.488186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:52.488632 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:52.488532    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:52.851246 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.851207    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k44hl" event={"ID":"9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd","Type":"ContainerStarted","Data":"edac466d36884d8b9859c10d73fecb1319c8ec620983c4ec01178fb108b731e1"}
Jul 28 10:23:52.853901 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.853882    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:23:52.854190 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.854166    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"5e8ac612f518987c6cf8d29da6c38b604d7c2277c634efea9e798cd36b60baa5"}
Jul 28 10:23:52.854243 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.854222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"3f58a845ab21a356b347858f38f5f7b0d8062de79820c7cd680ce4aa9ea49d1f"}
Jul 28 10:23:52.854299 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.854252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"1ebead8e9dcb2d00e91a3f5bc3630aedf193950911c58c880437329d3a7c44fc"}
Jul 28 10:23:52.855519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.855491    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q5h4z" event={"ID":"24a56284-b1d8-41d0-8f05-a14d507b6e4a","Type":"ContainerStarted","Data":"0e7b85079a3332ab8ebd2ab9b21ff26132dadad507881c73fd51024cd9719f86"}
Jul 28 10:23:52.856868 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.856844    2578 generic.go:356] "Generic (PLEG): container finished" podID="04b03a5e-2ba6-4e19-80f0-e24aea151a63" containerID="6d5087489fd94de34052d6101738b2f2dabfaed438d93da65d28165388837195" exitCode=0
Jul 28 10:23:52.856952 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.856925    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerDied","Data":"6d5087489fd94de34052d6101738b2f2dabfaed438d93da65d28165388837195"}
Jul 28 10:23:52.858217 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.858179    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-h55h6" event={"ID":"c7016e81-0f35-49e8-9319-c906df347a4f","Type":"ContainerStarted","Data":"dac9566c22e0e695a1139043699f228d45c2f4f96f729d1e7a90dae5c625b759"}
Jul 28 10:23:52.859671 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.859651    2578 generic.go:356] "Generic (PLEG): container finished" podID="e3fc6426ab408b4eb2e063772fac6b7e" containerID="acb8c6fce85bb5c6a686fb046b3799e039be273ec23640980f198c312e7820b7" exitCode=0
Jul 28 10:23:52.859758 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.859738    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" event={"ID":"e3fc6426ab408b4eb2e063772fac6b7e","Type":"ContainerDied","Data":"acb8c6fce85bb5c6a686fb046b3799e039be273ec23640980f198c312e7820b7"}
Jul 28 10:23:52.861146 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.861124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm" event={"ID":"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f","Type":"ContainerStarted","Data":"7e879bc76d3d114e897b6cddf24ad2312531dd5578bc2d0dea7099afe1728b04"}
Jul 28 10:23:52.863021 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.862999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cg6x9" event={"ID":"5d9439f1-1ac5-466a-bace-3b399fed2355","Type":"ContainerStarted","Data":"91a52c7e49c964fd62414bf9f8999a5dfe1c7a4a293c34d9a5593184667a9b15"}
Jul 28 10:23:52.870204 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.870160    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cvrqj" podStartSLOduration=2.830838076 podStartE2EDuration="23.870145533s" totalImagesPullingTime="21.039307457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:51.88042865 +0000 UTC m=+22.922520803" watchObservedRunningTime="2026-07-28 10:23:52.870145533 +0000 UTC m=+23.912237664"
Jul 28 10:23:52.870820 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.870796    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k44hl" podStartSLOduration=2.308805233 podStartE2EDuration="22.87078761s" totalImagesPullingTime="20.561982377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:52.869164025 +0000 UTC m=+23.911256179" watchObservedRunningTime="2026-07-28 10:23:52.87078761 +0000 UTC m=+23.912879762"
Jul 28 10:23:52.888093 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.888051    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-h55h6" podStartSLOduration=3.2216358290000002 podStartE2EDuration="23.888035658s" totalImagesPullingTime="20.666399829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:52.887456833 +0000 UTC m=+23.929548996" watchObservedRunningTime="2026-07-28 10:23:52.888035658 +0000 UTC m=+23.930127812"
Jul 28 10:23:52.904438 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.904393    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cg6x9" podStartSLOduration=3.245926238 podStartE2EDuration="23.90437906s" totalImagesPullingTime="20.658452822s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:52.904289333 +0000 UTC m=+23.946381489" watchObservedRunningTime="2026-07-28 10:23:52.90437906 +0000 UTC m=+23.946471212"
Jul 28 10:23:52.923408 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:52.923261    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-q5h4z" podStartSLOduration=2.328892967 podStartE2EDuration="22.923244974s" totalImagesPullingTime="20.594352007s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:52.922855597 +0000 UTC m=+23.964947750" watchObservedRunningTime="2026-07-28 10:23:52.923244974 +0000 UTC m=+23.965337126"
Jul 28 10:23:53.275321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.275288    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 10:23:53.398976 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.398716    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:23:53.275314298Z","UUID":"eaa66665-2e28-4119-bfdc-4d1f2168e1be","Handler":null,"Name":"","Endpoint":""}
Jul 28 10:23:53.400456 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.400433    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 10:23:53.400574 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.400479    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 10:23:53.489433 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.489361    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:53.489602 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:53.489491    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:53.489899 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.489879    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:53.490007 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:53.489980    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:53.868910 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.868822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" event={"ID":"e3fc6426ab408b4eb2e063772fac6b7e","Type":"ContainerStarted","Data":"eaec4ec0bc0a9971e0f0a6a2f16e97dca02c725e7bc7cfa8f01203c3d46d41c9"}
Jul 28 10:23:53.870799 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.870767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm" event={"ID":"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f","Type":"ContainerStarted","Data":"521e4effbfb000d4ff7964458d5a5f25129e89ee3a760ebedadecfbdab33b744"}
Jul 28 10:23:53.884769 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:53.884711    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-67.ec2.internal" podStartSLOduration=23.884692022 podStartE2EDuration="23.884692022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:53.884350731 +0000 UTC m=+24.926442883" watchObservedRunningTime="2026-07-28 10:23:53.884692022 +0000 UTC m=+24.926784177"
Jul 28 10:23:54.126856 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.126760    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:54.127529 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.127429    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:54.489179 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.489145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:54.489367 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:54.489299    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:54.876183 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.876102    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:23:54.876630 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.876562    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"d0a61c8bc1ebe970f296f681c02842fd9afb9200b6cf2733e9bd273d8414c3b8"}
Jul 28 10:23:54.878990 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.878958    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm" event={"ID":"613a23f3-d82b-4ba6-9dbe-2a6b2bd8709f","Type":"ContainerStarted","Data":"ba46fa6beaaf11e348151040a456d7b376fe51580ea58ff546b44a2e09a86716"}
Jul 28 10:23:54.879330 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.879302    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:54.879862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.879839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-h55h6"
Jul 28 10:23:54.926044 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:54.925991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcfhm" podStartSLOduration=2.059303917 podStartE2EDuration="24.925970335s" totalImagesPullingTime="22.866666418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:54.910649312 +0000 UTC m=+25.952741464" watchObservedRunningTime="2026-07-28 10:23:54.925970335 +0000 UTC m=+25.968062490"
Jul 28 10:23:55.488599 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:55.488563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:55.488783 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:55.488616    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:55.488783 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:55.488703    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:55.488896 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:55.488812    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:56.488384 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:56.488342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:56.488825 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:56.488477    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:57.489033 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.488797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:57.489779 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.488811    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:57.489779 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:57.489074    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:57.489779 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:57.489144    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:57.889853 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.889776    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:23:57.890172 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.890135    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"40b83af3de16c6b7d29bd3a5922185962950ca6a6df387fe133c5604c563a795"}
Jul 28 10:23:57.890531 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.890393    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:57.890531 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.890478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:57.890686 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.890606    2578 scope.go:117] "RemoveContainer" containerID="3bd79cb3a4d064b73aa30d0cac81272876cbd17fffbe3cc783ba80be59e16763"
Jul 28 10:23:57.892291 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.892240    2578 generic.go:356] "Generic (PLEG): container finished" podID="04b03a5e-2ba6-4e19-80f0-e24aea151a63" containerID="2c9c2512d5e9c0da4d9800ee3732def3c082e89115e8d72a58f92082e6c64d6e" exitCode=0
Jul 28 10:23:57.892291 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.892262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerDied","Data":"2c9c2512d5e9c0da4d9800ee3732def3c082e89115e8d72a58f92082e6c64d6e"}
Jul 28 10:23:57.907993 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:57.907966    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:58.489035 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.488850    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:23:58.489533 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:58.489108    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:23:58.898981 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.898957    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:23:58.899341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.899317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" event={"ID":"e3c17446-0ae7-4e7f-bbf6-669b2625cbed","Type":"ContainerStarted","Data":"a355d7ac9f2504311fe307903ae38146494ac58f6a151da7b7823818ab551387"}
Jul 28 10:23:58.899736 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.899718    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:58.901639 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.901611    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerStarted","Data":"9fbd30c82f5e3de9a20ab286e438aeaa64feadf380a8bdfdeb6c0eb98ce5c3cd"}
Jul 28 10:23:58.916399 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.916373    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:23:58.927925 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:58.927863    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn" podStartSLOduration=8.293783591 podStartE2EDuration="28.927846311s" totalImagesPullingTime="20.63406272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:23:58.926468973 +0000 UTC m=+29.968561126" watchObservedRunningTime="2026-07-28 10:23:58.927846311 +0000 UTC m=+29.969938465"
Jul 28 10:23:59.489711 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:59.489678    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:23:59.490153 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:59.489761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:23:59.490153 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:59.489795    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:23:59.490153 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:23:59.489865    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:23:59.905818 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:59.905728    2578 generic.go:356] "Generic (PLEG): container finished" podID="04b03a5e-2ba6-4e19-80f0-e24aea151a63" containerID="9fbd30c82f5e3de9a20ab286e438aeaa64feadf380a8bdfdeb6c0eb98ce5c3cd" exitCode=0
Jul 28 10:23:59.905982 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:23:59.905818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerDied","Data":"9fbd30c82f5e3de9a20ab286e438aeaa64feadf380a8bdfdeb6c0eb98ce5c3cd"}
Jul 28 10:24:00.488175 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:00.488135    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:00.488391 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:00.488292    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:01.488490 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:01.488460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:01.488859 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:01.488592    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:01.488859 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:01.488635    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:01.488859 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:01.488739    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:01.913184 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:01.913097    2578 generic.go:356] "Generic (PLEG): container finished" podID="04b03a5e-2ba6-4e19-80f0-e24aea151a63" containerID="0a294ffd061375077ad7eeb0a8f71b75f6159814562c43d33205ea9d93e62f8b" exitCode=0
Jul 28 10:24:01.913184 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:01.913158    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerDied","Data":"0a294ffd061375077ad7eeb0a8f71b75f6159814562c43d33205ea9d93e62f8b"}
Jul 28 10:24:02.178219 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:02.178182    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:02.178230    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:02.178251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178343    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178383    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178407    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178413    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs podName:c9311b88-8997-40d9-a0ea-5f5b5beca84f nodeName:}" failed. No retries permitted until 2026-07-28 10:24:34.178397004 +0000 UTC m=+65.220489135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs") pod "network-metrics-daemon-d5g2m" (UID: "c9311b88-8997-40d9-a0ea-5f5b5beca84f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178420    2578 projected.go:196] Error preparing data for projected volume kube-api-access-7cfdm for pod openshift-network-diagnostics/network-check-target-nsc54: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:24:02.178434 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178345    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:24:02.178734 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178470    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm podName:3d0ae63a-a197-421c-b2f6-f56f891ed32b nodeName:}" failed. No retries permitted until 2026-07-28 10:24:34.178454757 +0000 UTC m=+65.220546902 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7cfdm" (UniqueName: "kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm") pod "network-check-target-nsc54" (UID: "3d0ae63a-a197-421c-b2f6-f56f891ed32b") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:24:02.178734 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.178490    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret podName:081537ec-b941-46e9-b73e-f156ddb7585c nodeName:}" failed. No retries permitted until 2026-07-28 10:24:34.178480127 +0000 UTC m=+65.220572260 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret") pod "global-pull-secret-syncer-d4s8q" (UID: "081537ec-b941-46e9-b73e-f156ddb7585c") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:24:02.488765 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:02.488683    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:02.489204 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:02.488804    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:03.488305 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:03.488237    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:03.488305 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:03.488251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:03.488531 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:03.488424    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:03.488531 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:03.488513    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:04.488756 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:04.488724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:04.489152 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:04.488896    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:05.488544 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:05.488515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:05.488726 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:05.488515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:05.488726 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:05.488635    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:05.488726 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:05.488709    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:05.926171 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:05.926124    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerStarted","Data":"65fbca3dca23ebc11f9fd2a0ddeca9d1f70baa9de2931bca6e9e20a786ecb2fd"}
Jul 28 10:24:06.488409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:06.488364    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:06.488587 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:06.488482    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:06.931179 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:06.931140    2578 generic.go:356] "Generic (PLEG): container finished" podID="04b03a5e-2ba6-4e19-80f0-e24aea151a63" containerID="65fbca3dca23ebc11f9fd2a0ddeca9d1f70baa9de2931bca6e9e20a786ecb2fd" exitCode=0
Jul 28 10:24:06.931577 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:06.931204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerDied","Data":"65fbca3dca23ebc11f9fd2a0ddeca9d1f70baa9de2931bca6e9e20a786ecb2fd"}
Jul 28 10:24:07.488658 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:07.488620    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:07.488855 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:07.488626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:07.488855 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:07.488764    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:07.488855 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:07.488826    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:07.936964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:07.936927    2578 generic.go:356] "Generic (PLEG): container finished" podID="04b03a5e-2ba6-4e19-80f0-e24aea151a63" containerID="8985d31906ca3f8ea0e047c411a6421109e0d5a0b6056b5ebc9d7846d6535119" exitCode=0
Jul 28 10:24:07.937397 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:07.936996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerDied","Data":"8985d31906ca3f8ea0e047c411a6421109e0d5a0b6056b5ebc9d7846d6535119"}
Jul 28 10:24:08.488999 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:08.488952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:08.489206 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:08.489084    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:08.941888 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:08.941858    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dstdj" event={"ID":"04b03a5e-2ba6-4e19-80f0-e24aea151a63","Type":"ContainerStarted","Data":"94933b82d0a09750520268c54e4b3128378c760762399be562c32e568c3fa0e6"}
Jul 28 10:24:08.966056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:08.965987    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dstdj" podStartSLOduration=9.315018624 podStartE2EDuration="39.965970563s" totalImagesPullingTime="30.650951939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:08.965610537 +0000 UTC m=+40.007702691" watchObservedRunningTime="2026-07-28 10:24:08.965970563 +0000 UTC m=+40.008062715"
Jul 28 10:24:09.489459 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:09.489408    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:09.489670 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:09.489476    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:09.489670 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:09.489496    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:09.489670 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:09.489569    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:10.301079 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:10.300836    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d4s8q"]
Jul 28 10:24:10.301501 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:10.301172    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:10.301501 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:10.301308    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:10.304348 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:10.304318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d5g2m"]
Jul 28 10:24:10.304483 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:10.304457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:10.304594 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:10.304574    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:10.304860 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:10.304845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nsc54"]
Jul 28 10:24:10.304914 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:10.304904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:10.305002 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:10.304987    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:11.488618 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:11.488574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:11.489346 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:11.488685    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:11.489346 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:11.488737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:11.489346 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:11.488829    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:12.488325 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:12.488262    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:12.488478 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:12.488437    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d4s8q" podUID="081537ec-b941-46e9-b73e-f156ddb7585c"
Jul 28 10:24:13.488793 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:13.488757    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:13.489183 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:13.488902    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d5g2m" podUID="c9311b88-8997-40d9-a0ea-5f5b5beca84f"
Jul 28 10:24:13.489183 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:13.488951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:13.489183 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:13.489039    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nsc54" podUID="3d0ae63a-a197-421c-b2f6-f56f891ed32b"
Jul 28 10:24:14.321928 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.321898    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-67.ec2.internal" event="NodeReady"
Jul 28 10:24:14.322115 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.322045    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 10:24:14.381036 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.380958    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-k4f7h"]
Jul 28 10:24:14.391313 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.391263    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.393051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.393030    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gsj9h"]
Jul 28 10:24:14.394502 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.394475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8f5cv\""
Jul 28 10:24:14.394715 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.394693    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 10:24:14.395111 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.395099    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 10:24:14.403398 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.403376    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k4f7h"]
Jul 28 10:24:14.403532 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.403481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.406560 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.406536    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-9dkdj\""
Jul 28 10:24:14.407712 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.407684    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 10:24:14.408749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.408239    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 10:24:14.409484 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.409182    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 10:24:14.409484 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.409361    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gsj9h"]
Jul 28 10:24:14.412607 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.412578    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhdw"]
Jul 28 10:24:14.423890 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.423867    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.426310 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.426286    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 10:24:14.426487 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.426463    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhdw"]
Jul 28 10:24:14.426612 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.426590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-kn9mk\""
Jul 28 10:24:14.426675 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.426590    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 10:24:14.426778 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.426759    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 10:24:14.426882 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.426780    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 10:24:14.488967 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.488935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:14.491530 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.491508    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:24:14.571808 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.571769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c7e810-2dcf-4345-8622-13f6d2e2dbef-tmp-dir\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.572002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.571816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11149244-998e-4872-8941-9fabad25d0a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.572002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.571884    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn497\" (UniqueName: \"kubernetes.io/projected/d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b-kube-api-access-tn497\") pod \"ingress-canary-gsj9h\" (UID: \"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b\") " pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.572002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.571950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37c7e810-2dcf-4345-8622-13f6d2e2dbef-config-volume\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.572002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.571982    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/11149244-998e-4872-8941-9fabad25d0a4-data-volume\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.572173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.572032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11149244-998e-4872-8941-9fabad25d0a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.572173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.572059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdrv9\" (UniqueName: \"kubernetes.io/projected/11149244-998e-4872-8941-9fabad25d0a4-kube-api-access-tdrv9\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.572173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.572107    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/11149244-998e-4872-8941-9fabad25d0a4-crio-socket\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.572173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.572134    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q99v\" (UniqueName: \"kubernetes.io/projected/37c7e810-2dcf-4345-8622-13f6d2e2dbef-kube-api-access-7q99v\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.572173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.572151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b-cert\") pod \"ingress-canary-gsj9h\" (UID: \"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b\") " pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.572358 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.572182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37c7e810-2dcf-4345-8622-13f6d2e2dbef-metrics-tls\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.673137 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11149244-998e-4872-8941-9fabad25d0a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdrv9\" (UniqueName: \"kubernetes.io/projected/11149244-998e-4872-8941-9fabad25d0a4-kube-api-access-tdrv9\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673175    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/11149244-998e-4872-8941-9fabad25d0a4-crio-socket\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q99v\" (UniqueName: \"kubernetes.io/projected/37c7e810-2dcf-4345-8622-13f6d2e2dbef-kube-api-access-7q99v\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b-cert\") pod \"ingress-canary-gsj9h\" (UID: \"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b\") " pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37c7e810-2dcf-4345-8622-13f6d2e2dbef-metrics-tls\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673244    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c7e810-2dcf-4345-8622-13f6d2e2dbef-tmp-dir\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.673259 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673259    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11149244-998e-4872-8941-9fabad25d0a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.673609 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn497\" (UniqueName: \"kubernetes.io/projected/d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b-kube-api-access-tn497\") pod \"ingress-canary-gsj9h\" (UID: \"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b\") " pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.673609 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37c7e810-2dcf-4345-8622-13f6d2e2dbef-config-volume\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.673790 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673759    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/11149244-998e-4872-8941-9fabad25d0a4-data-volume\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.673853 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673826    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37c7e810-2dcf-4345-8622-13f6d2e2dbef-tmp-dir\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.674007 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673976    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/11149244-998e-4872-8941-9fabad25d0a4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.674113 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.673985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/37c7e810-2dcf-4345-8622-13f6d2e2dbef-config-volume\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.674113 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.674069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/11149244-998e-4872-8941-9fabad25d0a4-data-volume\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.674181 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.674120    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/11149244-998e-4872-8941-9fabad25d0a4-crio-socket\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.677289 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.677243    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/37c7e810-2dcf-4345-8622-13f6d2e2dbef-metrics-tls\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.677394 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.677373    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/11149244-998e-4872-8941-9fabad25d0a4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.677494 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.677479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b-cert\") pod \"ingress-canary-gsj9h\" (UID: \"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b\") " pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.681898 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.681875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn497\" (UniqueName: \"kubernetes.io/projected/d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b-kube-api-access-tn497\") pod \"ingress-canary-gsj9h\" (UID: \"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b\") " pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.682165 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.682140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdrv9\" (UniqueName: \"kubernetes.io/projected/11149244-998e-4872-8941-9fabad25d0a4-kube-api-access-tdrv9\") pod \"insights-runtime-extractor-wkhdw\" (UID: \"11149244-998e-4872-8941-9fabad25d0a4\") " pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.682286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.682194    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q99v\" (UniqueName: \"kubernetes.io/projected/37c7e810-2dcf-4345-8622-13f6d2e2dbef-kube-api-access-7q99v\") pod \"dns-default-k4f7h\" (UID: \"37c7e810-2dcf-4345-8622-13f6d2e2dbef\") " pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.702107 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.702062    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:14.717224 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.717194    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gsj9h"
Jul 28 10:24:14.733400 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.733373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-wkhdw"
Jul 28 10:24:14.935400 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.935307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gsj9h"]
Jul 28 10:24:14.938386 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.938362    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-k4f7h"]
Jul 28 10:24:14.939831 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:14.939806    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3cc6e7b_8e03_45b4_8853_dd964b1b2f1b.slice/crio-fd9f2ca0ac000de633e94be7ea2bf361320b6662bb8eaa322f8c8800572e23a3 WatchSource:0}: Error finding container fd9f2ca0ac000de633e94be7ea2bf361320b6662bb8eaa322f8c8800572e23a3: Status 404 returned error can't find the container with id fd9f2ca0ac000de633e94be7ea2bf361320b6662bb8eaa322f8c8800572e23a3
Jul 28 10:24:14.941583 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.941479    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-wkhdw"]
Jul 28 10:24:14.941805 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:14.941754    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c7e810_2dcf_4345_8622_13f6d2e2dbef.slice/crio-385189e799b3d3ff06042a4c81898d31497f3a5afd7f74a336a8829ece60d04a WatchSource:0}: Error finding container 385189e799b3d3ff06042a4c81898d31497f3a5afd7f74a336a8829ece60d04a: Status 404 returned error can't find the container with id 385189e799b3d3ff06042a4c81898d31497f3a5afd7f74a336a8829ece60d04a
Jul 28 10:24:14.947947 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:14.947916    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11149244_998e_4872_8941_9fabad25d0a4.slice/crio-0b299b20f36f4fcdc4c1da26ac956a54f759d559c4608ea8a77b2f34fe877497 WatchSource:0}: Error finding container 0b299b20f36f4fcdc4c1da26ac956a54f759d559c4608ea8a77b2f34fe877497: Status 404 returned error can't find the container with id 0b299b20f36f4fcdc4c1da26ac956a54f759d559c4608ea8a77b2f34fe877497
Jul 28 10:24:14.956237 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.956202    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhdw" event={"ID":"11149244-998e-4872-8941-9fabad25d0a4","Type":"ContainerStarted","Data":"0b299b20f36f4fcdc4c1da26ac956a54f759d559c4608ea8a77b2f34fe877497"}
Jul 28 10:24:14.957287 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.957249    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gsj9h" event={"ID":"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b","Type":"ContainerStarted","Data":"fd9f2ca0ac000de633e94be7ea2bf361320b6662bb8eaa322f8c8800572e23a3"}
Jul 28 10:24:14.958341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:14.958304    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k4f7h" event={"ID":"37c7e810-2dcf-4345-8622-13f6d2e2dbef","Type":"ContainerStarted","Data":"385189e799b3d3ff06042a4c81898d31497f3a5afd7f74a336a8829ece60d04a"}
Jul 28 10:24:15.489475 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.488860    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:15.489475 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.488899    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:15.493089 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.493060    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8src\""
Jul 28 10:24:15.493235 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.493119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:24:15.493349 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.493331    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:24:15.493485 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.493467    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mqqnj\""
Jul 28 10:24:15.493593 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:15.493572    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:24:16.525793 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.524792    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"]
Jul 28 10:24:16.550183 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.550148    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"]
Jul 28 10:24:16.550370 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.550316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.553170 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.552893    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 10:24:16.553170 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.552917    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-tmsv7\""
Jul 28 10:24:16.553170 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.553141    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 10:24:16.553780 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.553761    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 10:24:16.554157 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.554138    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 10:24:16.554579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.554551    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 10:24:16.693247 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.693208    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbjps\" (UniqueName: \"kubernetes.io/projected/1d917317-ff26-4bfd-9da3-13d4d68761cc-kube-api-access-sbjps\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.693453 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.693300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d917317-ff26-4bfd-9da3-13d4d68761cc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.693453 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.693353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d917317-ff26-4bfd-9da3-13d4d68761cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.693453 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.693381    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d917317-ff26-4bfd-9da3-13d4d68761cc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.795417 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.793843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d917317-ff26-4bfd-9da3-13d4d68761cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.795417 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.793901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d917317-ff26-4bfd-9da3-13d4d68761cc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.795417 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.793954    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbjps\" (UniqueName: \"kubernetes.io/projected/1d917317-ff26-4bfd-9da3-13d4d68761cc-kube-api-access-sbjps\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.795417 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.794001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d917317-ff26-4bfd-9da3-13d4d68761cc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.795417 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.795341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1d917317-ff26-4bfd-9da3-13d4d68761cc-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.798121 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.798094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1d917317-ff26-4bfd-9da3-13d4d68761cc-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.798121 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.798110    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1d917317-ff26-4bfd-9da3-13d4d68761cc-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.804425 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.804397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbjps\" (UniqueName: \"kubernetes.io/projected/1d917317-ff26-4bfd-9da3-13d4d68761cc-kube-api-access-sbjps\") pod \"prometheus-operator-84d69f794d-ctxfh\" (UID: \"1d917317-ff26-4bfd-9da3-13d4d68761cc\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:16.868385 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:16.868348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"
Jul 28 10:24:21.868715 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:21.868459    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ctxfh"]
Jul 28 10:24:21.872968 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:21.872940    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d917317_ff26_4bfd_9da3_13d4d68761cc.slice/crio-57570e505e1b1f760a44fc2757b5197b59fca98eb7630b6ca1bbe36b5ff0c81e WatchSource:0}: Error finding container 57570e505e1b1f760a44fc2757b5197b59fca98eb7630b6ca1bbe36b5ff0c81e: Status 404 returned error can't find the container with id 57570e505e1b1f760a44fc2757b5197b59fca98eb7630b6ca1bbe36b5ff0c81e
Jul 28 10:24:21.979884 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:21.979844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k4f7h" event={"ID":"37c7e810-2dcf-4345-8622-13f6d2e2dbef","Type":"ContainerStarted","Data":"e15d95a6831d530b0691266505aa2ecb10416a0ca93af25b4bc56f29aee76e6f"}
Jul 28 10:24:21.981407 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:21.981343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gsj9h" event={"ID":"d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b","Type":"ContainerStarted","Data":"cabe9901b7ec82e64f444a395171f79db41aa7f37f3013a310867d7e1ceaf692"}
Jul 28 10:24:21.982626 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:21.982600    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh" event={"ID":"1d917317-ff26-4bfd-9da3-13d4d68761cc","Type":"ContainerStarted","Data":"57570e505e1b1f760a44fc2757b5197b59fca98eb7630b6ca1bbe36b5ff0c81e"}
Jul 28 10:24:21.996470 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:21.996418    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gsj9h" podStartSLOduration=1.208553445 podStartE2EDuration="7.996399806s" totalImagesPullingTime="6.787846361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:21.995124324 +0000 UTC m=+53.037216477" watchObservedRunningTime="2026-07-28 10:24:21.996399806 +0000 UTC m=+53.038491938"
Jul 28 10:24:22.989242 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:22.989200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-k4f7h" event={"ID":"37c7e810-2dcf-4345-8622-13f6d2e2dbef","Type":"ContainerStarted","Data":"d160a9e552492013c2de5a4a0128508730811c537c2b9630c170b9a1d40d28ab"}
Jul 28 10:24:22.990123 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:22.990058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:22.992286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:22.992230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhdw" event={"ID":"11149244-998e-4872-8941-9fabad25d0a4","Type":"ContainerStarted","Data":"f740e2632f2ca645bf3fa1d5bad5e051b40dde1c9a332ccefc1d4d2a69cb408d"}
Jul 28 10:24:23.008620 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:23.008077    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-k4f7h" podStartSLOduration=2.221940733 podStartE2EDuration="9.008052885s" totalImagesPullingTime="6.786112152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:23.007474387 +0000 UTC m=+54.049566565" watchObservedRunningTime="2026-07-28 10:24:23.008052885 +0000 UTC m=+54.050145052"
Jul 28 10:24:26.003416 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.003369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh" event={"ID":"1d917317-ff26-4bfd-9da3-13d4d68761cc","Type":"ContainerStarted","Data":"84300fe495e211b9309d0090decbe25d5c819cc55a9d0be31b954aab45c1d6a6"}
Jul 28 10:24:26.003876 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.003480    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh" event={"ID":"1d917317-ff26-4bfd-9da3-13d4d68761cc","Type":"ContainerStarted","Data":"1688bba72360ba21730c4376000f5a1dcfb26384edb4d99531dddffea543f51f"}
Jul 28 10:24:26.005598 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.005561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhdw" event={"ID":"11149244-998e-4872-8941-9fabad25d0a4","Type":"ContainerStarted","Data":"88935f4cdc90fa23509fb15f9b95937156e24a6e17ffe5918ad66d0eb162df19"}
Jul 28 10:24:26.026610 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.026559    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-ctxfh" podStartSLOduration=6.081807443 podStartE2EDuration="10.026541444s" totalImagesPullingTime="3.944734001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:26.026151958 +0000 UTC m=+57.068244173" watchObservedRunningTime="2026-07-28 10:24:26.026541444 +0000 UTC m=+57.068633596"
Jul 28 10:24:26.038898 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.038858    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-bb7db"]
Jul 28 10:24:26.040779 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.040761    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:26.043019 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.042987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 10:24:26.043173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.043155    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 10:24:26.043247 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.043232    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-q9z8d\""
Jul 28 10:24:26.051141 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.051113    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-bb7db"]
Jul 28 10:24:26.066605 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.066565    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9vh\" (UniqueName: \"kubernetes.io/projected/3bb5f1bb-5a0c-46da-a128-968330d00d73-kube-api-access-5g9vh\") pod \"downloads-7c97dc4555-bb7db\" (UID: \"3bb5f1bb-5a0c-46da-a128-968330d00d73\") " pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:26.167856 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.167767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9vh\" (UniqueName: \"kubernetes.io/projected/3bb5f1bb-5a0c-46da-a128-968330d00d73-kube-api-access-5g9vh\") pod \"downloads-7c97dc4555-bb7db\" (UID: \"3bb5f1bb-5a0c-46da-a128-968330d00d73\") " pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:26.178162 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.178138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9vh\" (UniqueName: \"kubernetes.io/projected/3bb5f1bb-5a0c-46da-a128-968330d00d73-kube-api-access-5g9vh\") pod \"downloads-7c97dc4555-bb7db\" (UID: \"3bb5f1bb-5a0c-46da-a128-968330d00d73\") " pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:26.350702 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.350662    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:26.503248 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:26.503144    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-bb7db"]
Jul 28 10:24:26.506978 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:26.506947    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb5f1bb_5a0c_46da_a128_968330d00d73.slice/crio-fe4ef28d7f5feb636095d411e3c5675ec1e58d3ff742f768b9c0c3efa007e9cd WatchSource:0}: Error finding container fe4ef28d7f5feb636095d411e3c5675ec1e58d3ff742f768b9c0c3efa007e9cd: Status 404 returned error can't find the container with id fe4ef28d7f5feb636095d411e3c5675ec1e58d3ff742f768b9c0c3efa007e9cd
Jul 28 10:24:27.010049 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.010006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-bb7db" event={"ID":"3bb5f1bb-5a0c-46da-a128-968330d00d73","Type":"ContainerStarted","Data":"fe4ef28d7f5feb636095d411e3c5675ec1e58d3ff742f768b9c0c3efa007e9cd"}
Jul 28 10:24:27.879702 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.879664    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"]
Jul 28 10:24:27.881821 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.881792    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:27.887229 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.887197    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 10:24:27.887553 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.887521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rtmtb\""
Jul 28 10:24:27.887712 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.887698    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 10:24:27.903369 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.903339    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"]
Jul 28 10:24:27.933365 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.933329    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sf288"]
Jul 28 10:24:27.935719 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.935691    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.939243 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.938693    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 10:24:27.939243 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.939049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qpdwx\""
Jul 28 10:24:27.939474 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.939435    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 10:24:27.942096 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.942074    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 10:24:27.981725 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-root\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.981895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-wtmp\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.981895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981806    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jrd2\" (UniqueName: \"kubernetes.io/projected/4951a454-2c35-42c0-a2b3-03e34925ce23-kube-api-access-2jrd2\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.981895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.981895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981886    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-tls\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.982127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981946    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4951a454-2c35-42c0-a2b3-03e34925ce23-metrics-client-ca\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.982127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.981983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:27.982127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.982030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/107932d8-9f04-42e6-8db2-8987be5feb87-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:27.982127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.982058    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.982127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.982088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lqb\" (UniqueName: \"kubernetes.io/projected/107932d8-9f04-42e6-8db2-8987be5feb87-kube-api-access-62lqb\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:27.982127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.982125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-textfile\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:27.982601 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.982161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:27.982601 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:27.982226    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-sys\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082713    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-sys\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-root\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-wtmp\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jrd2\" (UniqueName: \"kubernetes.io/projected/4951a454-2c35-42c0-a2b3-03e34925ce23-kube-api-access-2jrd2\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082847    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-tls\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4951a454-2c35-42c0-a2b3-03e34925ce23-metrics-client-ca\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082945    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.082996    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/107932d8-9f04-42e6-8db2-8987be5feb87-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.083031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.083064    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lqb\" (UniqueName: \"kubernetes.io/projected/107932d8-9f04-42e6-8db2-8987be5feb87-kube-api-access-62lqb\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.083102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-textfile\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.083499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-textfile\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:28.084032    2578 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 10:24:28.084752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.084039    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-wtmp\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:28.084105    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-tls podName:107932d8-9f04-42e6-8db2-8987be5feb87 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:28.58408883 +0000 UTC m=+59.626180965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-mpm5v" (UID: "107932d8-9f04-42e6-8db2-8987be5feb87") : secret "openshift-state-metrics-tls" not found
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:28.084429    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:28.084482    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-tls podName:4951a454-2c35-42c0-a2b3-03e34925ce23 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:28.584465035 +0000 UTC m=+59.626557182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-tls") pod "node-exporter-sf288" (UID: "4951a454-2c35-42c0-a2b3-03e34925ce23") : secret "node-exporter-tls" not found
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.084528    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-sys\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.084572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4951a454-2c35-42c0-a2b3-03e34925ce23-root\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.084664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4951a454-2c35-42c0-a2b3-03e34925ce23-metrics-client-ca\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.084741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-accelerators-collector-config\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.086090 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.084748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/107932d8-9f04-42e6-8db2-8987be5feb87-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.087564 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.087529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.089479 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.089453    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.105346 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.105258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lqb\" (UniqueName: \"kubernetes.io/projected/107932d8-9f04-42e6-8db2-8987be5feb87-kube-api-access-62lqb\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.108798 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.108767    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jrd2\" (UniqueName: \"kubernetes.io/projected/4951a454-2c35-42c0-a2b3-03e34925ce23-kube-api-access-2jrd2\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.588062 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.588010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-tls\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.588255 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.588068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.591082 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.591049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4951a454-2c35-42c0-a2b3-03e34925ce23-node-exporter-tls\") pod \"node-exporter-sf288\" (UID: \"4951a454-2c35-42c0-a2b3-03e34925ce23\") " pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:28.592576 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.592553    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/107932d8-9f04-42e6-8db2-8987be5feb87-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-mpm5v\" (UID: \"107932d8-9f04-42e6-8db2-8987be5feb87\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.799895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.799862    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"
Jul 28 10:24:28.856242 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:28.856159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sf288"
Jul 28 10:24:29.021544 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.021510    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:24:29.025997 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.025968    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.030970 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.030576    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 10:24:29.030970 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.030597    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 10:24:29.030970 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.030851    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.033647    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.033880    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.034091    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.034308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fwhw9\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.034539    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.034846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 10:24:29.036450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.035059    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 10:24:29.041897 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.041863    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:24:29.094073 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094088    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094142    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094197    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-web-config\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094258    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-config-out\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094373    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnw89\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-kube-api-access-qnw89\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094441    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094458    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.094517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.094473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195069 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195030    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195294 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195084    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195294 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-web-config\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195294 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195143    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195294 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-config-out\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195294 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195224    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195315    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnw89\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-kube-api-access-qnw89\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195394    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.195947 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.195550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.197062 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.197034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.197465 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.197442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.197986 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.197549    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.198766 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.198719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.199401 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.199301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.199825 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.199801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-config-out\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.200623 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.200600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.201386 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.201344    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-web-config\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.201484 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.201446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.201813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.201790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.202113 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.202093    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.202603 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.202584    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-config-volume\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.204803 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.204782    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnw89\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-kube-api-access-qnw89\") pod \"alertmanager-main-0\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.341552 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.341521    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fwhw9\""
Jul 28 10:24:29.349771 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:29.349726    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:24:29.943150 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:29.943110    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4951a454_2c35_42c0_a2b3_03e34925ce23.slice/crio-67bb16383f9b5668771f0eda87af1d859061bdeed09edebfbae756ce20be681c WatchSource:0}: Error finding container 67bb16383f9b5668771f0eda87af1d859061bdeed09edebfbae756ce20be681c: Status 404 returned error can't find the container with id 67bb16383f9b5668771f0eda87af1d859061bdeed09edebfbae756ce20be681c
Jul 28 10:24:30.029231 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:30.029150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf288" event={"ID":"4951a454-2c35-42c0-a2b3-03e34925ce23","Type":"ContainerStarted","Data":"67bb16383f9b5668771f0eda87af1d859061bdeed09edebfbae756ce20be681c"}
Jul 28 10:24:30.110703 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:30.110664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:24:30.119981 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:30.119943    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v"]
Jul 28 10:24:30.223091 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:30.223004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc5306a_0689_4813_a1a3_4bca52929958.slice/crio-3e709d930e29b43201aa39e3d444382efdb454249d94aeca645a9b6349a479c9 WatchSource:0}: Error finding container 3e709d930e29b43201aa39e3d444382efdb454249d94aeca645a9b6349a479c9: Status 404 returned error can't find the container with id 3e709d930e29b43201aa39e3d444382efdb454249d94aeca645a9b6349a479c9
Jul 28 10:24:30.936419 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:30.935873    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jmrkn"
Jul 28 10:24:31.045142 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.045098    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v" event={"ID":"107932d8-9f04-42e6-8db2-8987be5feb87","Type":"ContainerStarted","Data":"2ea0cfa6cb3618fbc3e6cef296d80bdcb4dcdf04d351c5f032e21198f51f8f27"}
Jul 28 10:24:31.045357 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.045216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v" event={"ID":"107932d8-9f04-42e6-8db2-8987be5feb87","Type":"ContainerStarted","Data":"926f2c68390323d4f8979319d0adb09dda20b910f0ca890a593a3320cf87e7b4"}
Jul 28 10:24:31.045357 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.045238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v" event={"ID":"107932d8-9f04-42e6-8db2-8987be5feb87","Type":"ContainerStarted","Data":"8f076fb232dd2e24872689ff8ac788633c2c5f0aad26d32c349f05f8b5dc6d2e"}
Jul 28 10:24:31.047210 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.047163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"3e709d930e29b43201aa39e3d444382efdb454249d94aeca645a9b6349a479c9"}
Jul 28 10:24:31.049802 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.049766    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-wkhdw" event={"ID":"11149244-998e-4872-8941-9fabad25d0a4","Type":"ContainerStarted","Data":"22184ad2634c2cb8dfa6f33f99ea6e238e5cf34058d6a6daf191d6f5a3d73432"}
Jul 28 10:24:31.068563 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.068507    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-wkhdw" podStartSLOduration=1.9345129079999999 podStartE2EDuration="17.068491565s" totalImagesPullingTime="15.133978657s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:24:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:31.068445849 +0000 UTC m=+62.110538004" watchObservedRunningTime="2026-07-28 10:24:31.068491565 +0000 UTC m=+62.110583696"
Jul 28 10:24:31.967300 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.966746    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-845c87956d-n52dk"]
Jul 28 10:24:31.969880 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.969858    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:31.972598 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.972570    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 10:24:31.972960 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.972945    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 10:24:31.973995 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.973470    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 10:24:31.973995 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.973476    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-j5kjh\""
Jul 28 10:24:31.973995 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.973529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 10:24:31.973995 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.973716    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 10:24:31.979801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:31.979778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845c87956d-n52dk"]
Jul 28 10:24:32.020610 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.020465    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-oauth-config\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.020610 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.020506    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-console-config\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.020610 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.020552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppcl\" (UniqueName: \"kubernetes.io/projected/474a6af3-842d-47aa-9360-fac0e6d87829-kube-api-access-sppcl\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.020858 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.020610    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-serving-cert\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.020858 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.020704    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-service-ca\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.020858 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.020773    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-oauth-serving-cert\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.054543 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.054510    2578 generic.go:356] "Generic (PLEG): container finished" podID="4951a454-2c35-42c0-a2b3-03e34925ce23" containerID="acc37dcb8d579d57482265956d79d25ba3eb050c4a0ecd6df88cdc3dd4ccbd1e" exitCode=0
Jul 28 10:24:32.054706 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.054596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf288" event={"ID":"4951a454-2c35-42c0-a2b3-03e34925ce23","Type":"ContainerDied","Data":"acc37dcb8d579d57482265956d79d25ba3eb050c4a0ecd6df88cdc3dd4ccbd1e"}
Jul 28 10:24:32.056819 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.056788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v" event={"ID":"107932d8-9f04-42e6-8db2-8987be5feb87","Type":"ContainerStarted","Data":"cdc2545f57b1634c9e2e1d6a01049320729b92700769072912f3fa306e2ede43"}
Jul 28 10:24:32.059083 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.059037    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="38341bebbaf3a7092245c4fbad458f65f8e1415d6eff052ba6564121d8c59591" exitCode=0
Jul 28 10:24:32.059172 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.059154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"38341bebbaf3a7092245c4fbad458f65f8e1415d6eff052ba6564121d8c59591"}
Jul 28 10:24:32.095650 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.095584    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-mpm5v" podStartSLOduration=3.700388675 podStartE2EDuration="5.095562679s" totalImagesPullingTime="1.395174004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:32.095365776 +0000 UTC m=+63.137457929" watchObservedRunningTime="2026-07-28 10:24:32.095562679 +0000 UTC m=+63.137654834"
Jul 28 10:24:32.122059 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-service-ca\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.122173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-oauth-serving-cert\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.122458 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-oauth-config\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.122566 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122489    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-console-config\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.122688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppcl\" (UniqueName: \"kubernetes.io/projected/474a6af3-842d-47aa-9360-fac0e6d87829-kube-api-access-sppcl\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.122749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122706    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-serving-cert\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.123400 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.122999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-service-ca\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.123685 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.123656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-console-config\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.124316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.124285    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-oauth-serving-cert\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.126679 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.126643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-oauth-config\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.130781 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.130750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-serving-cert\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.133291 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.133248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppcl\" (UniqueName: \"kubernetes.io/projected/474a6af3-842d-47aa-9360-fac0e6d87829-kube-api-access-sppcl\") pod \"console-845c87956d-n52dk\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") " pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.294346 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.294302    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:32.447987 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.447898    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-845c87956d-n52dk"]
Jul 28 10:24:32.451402 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:32.451360    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474a6af3_842d_47aa_9360_fac0e6d87829.slice/crio-bf855a67e924e03f7f2b9cd6e6f7d06f84b889d658c81560f29fb870c092bc4c WatchSource:0}: Error finding container bf855a67e924e03f7f2b9cd6e6f7d06f84b889d658c81560f29fb870c092bc4c: Status 404 returned error can't find the container with id bf855a67e924e03f7f2b9cd6e6f7d06f84b889d658c81560f29fb870c092bc4c
Jul 28 10:24:32.648933 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.648895    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"]
Jul 28 10:24:32.651063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.651037    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:32.653592 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.653566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 10:24:32.653731 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.653664    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-q5v8c\""
Jul 28 10:24:32.659521 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.659127    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"]
Jul 28 10:24:32.728861 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.728758    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6cwn\" (UniqueName: \"kubernetes.io/projected/bf3517c1-cacd-4ee7-af11-65d55097e2d5-kube-api-access-c6cwn\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:32.729038 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.728945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf3517c1-cacd-4ee7-af11-65d55097e2d5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:32.830485 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.830436    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf3517c1-cacd-4ee7-af11-65d55097e2d5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:32.830672 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.830543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6cwn\" (UniqueName: \"kubernetes.io/projected/bf3517c1-cacd-4ee7-af11-65d55097e2d5-kube-api-access-c6cwn\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:32.830672 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:32.830602    2578 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 28 10:24:32.830788 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:24:32.830675    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf3517c1-cacd-4ee7-af11-65d55097e2d5-monitoring-plugin-cert podName:bf3517c1-cacd-4ee7-af11-65d55097e2d5 nodeName:}" failed. No retries permitted until 2026-07-28 10:24:33.330655882 +0000 UTC m=+64.372748034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/bf3517c1-cacd-4ee7-af11-65d55097e2d5-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-jd9cw" (UID: "bf3517c1-cacd-4ee7-af11-65d55097e2d5") : secret "monitoring-plugin-cert" not found
Jul 28 10:24:32.841672 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:32.841634    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6cwn\" (UniqueName: \"kubernetes.io/projected/bf3517c1-cacd-4ee7-af11-65d55097e2d5-kube-api-access-c6cwn\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:33.065467 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.065261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845c87956d-n52dk" event={"ID":"474a6af3-842d-47aa-9360-fac0e6d87829","Type":"ContainerStarted","Data":"bf855a67e924e03f7f2b9cd6e6f7d06f84b889d658c81560f29fb870c092bc4c"}
Jul 28 10:24:33.068191 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.068152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf288" event={"ID":"4951a454-2c35-42c0-a2b3-03e34925ce23","Type":"ContainerStarted","Data":"97e427cddb3f472ca4c6a9252d3ac2665450711021cec036c1a1b5514daf74c8"}
Jul 28 10:24:33.068380 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.068230    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sf288" event={"ID":"4951a454-2c35-42c0-a2b3-03e34925ce23","Type":"ContainerStarted","Data":"6ef2aead1edef36a9be02254ac1eddd082792f6a0b4b2cab2dcb2bb3d34e5f89"}
Jul 28 10:24:33.088316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.088231    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sf288" podStartSLOduration=4.744385391 podStartE2EDuration="6.088214712s" totalImagesPullingTime="1.216170564s" totalInitContainerRuntime="127.658757ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:33.08720796 +0000 UTC m=+64.129300175" watchObservedRunningTime="2026-07-28 10:24:33.088214712 +0000 UTC m=+64.130306864"
Jul 28 10:24:33.336204 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.336119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf3517c1-cacd-4ee7-af11-65d55097e2d5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:33.339025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.338992    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bf3517c1-cacd-4ee7-af11-65d55097e2d5-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-jd9cw\" (UID: \"bf3517c1-cacd-4ee7-af11-65d55097e2d5\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:33.564455 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.564421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:34.000655 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:33.999743    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-k4f7h"
Jul 28 10:24:34.058814 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.058762    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"]
Jul 28 10:24:34.061984 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:34.061926    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3517c1_cacd_4ee7_af11_65d55097e2d5.slice/crio-e8335b0577e2140e48c0536b175b7083dbf3f0611dcecf81d9487aa262f767fa WatchSource:0}: Error finding container e8335b0577e2140e48c0536b175b7083dbf3f0611dcecf81d9487aa262f767fa: Status 404 returned error can't find the container with id e8335b0577e2140e48c0536b175b7083dbf3f0611dcecf81d9487aa262f767fa
Jul 28 10:24:34.084379 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.084328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"50b0201fd67e40766be161fce31401fadc7f761b8e371fc8dfd2dae83c2c389a"}
Jul 28 10:24:34.087721 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.087623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw" event={"ID":"bf3517c1-cacd-4ee7-af11-65d55097e2d5","Type":"ContainerStarted","Data":"e8335b0577e2140e48c0536b175b7083dbf3f0611dcecf81d9487aa262f767fa"}
Jul 28 10:24:34.218643 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.216500    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:24:34.220060 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.220035    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.224342 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.222805    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 10:24:34.224342 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.222903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 10:24:34.224342 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.223809    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 10:24:34.224342 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.223912    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 10:24:34.224342 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.224049    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 10:24:34.229660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.227656    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 10:24:34.229660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.227672    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 10:24:34.229660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.227834    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 10:24:34.229660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.227853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 10:24:34.229660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.228094    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 10:24:34.232029 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.230785    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 10:24:34.232029 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.231108    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lfjmt\""
Jul 28 10:24:34.232029 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.231344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 10:24:34.233420 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.233397    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ao6el0591c0sv\""
Jul 28 10:24:34.234322 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.234045    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 10:24:34.240004 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.234436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 10:24:34.240004 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.238139    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:24:34.244806 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.244781    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 10:24:34.248077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.248038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:34.248214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.248119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:34.248214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.248197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:34.251173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.251139    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:24:34.251482 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.251451    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:24:34.251579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.251483    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:24:34.263209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.263176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c9311b88-8997-40d9-a0ea-5f5b5beca84f-metrics-certs\") pod \"network-metrics-daemon-d5g2m\" (UID: \"c9311b88-8997-40d9-a0ea-5f5b5beca84f\") " pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:34.263595 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.263573    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:24:34.264444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.264417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/081537ec-b941-46e9-b73e-f156ddb7585c-original-pull-secret\") pod \"global-pull-secret-syncer-d4s8q\" (UID: \"081537ec-b941-46e9-b73e-f156ddb7585c\") " pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:34.272540 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.272508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfdm\" (UniqueName: \"kubernetes.io/projected/3d0ae63a-a197-421c-b2f6-f56f891ed32b-kube-api-access-7cfdm\") pod \"network-check-target-nsc54\" (UID: \"3d0ae63a-a197-421c-b2f6-f56f891ed32b\") " pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:34.297994 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.297732    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d4s8q"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350075    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78lp\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-kube-api-access-d78lp\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350158    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350194    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350216    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-config\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350340    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350387    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350536    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.351192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350609    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.352306 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350645    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.352306 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350685    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.352306 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.352306 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.350747    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.410980 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.410747    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8src\""
Jul 28 10:24:34.413716 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.413690    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-mqqnj\""
Jul 28 10:24:34.417638 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.417507    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d5g2m"
Jul 28 10:24:34.422798 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.422562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:34.453878 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.453878 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453687    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.453878 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78lp\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-kube-api-access-d78lp\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.453878 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453743    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.453878 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453876    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453957    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-config\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.453987    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454054    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454131    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454184    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454863 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454317    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454863 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454863 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454863 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454422    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.454863 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.454562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.455113 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.455098    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.455336 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.455230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.455548 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.455523    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.459507 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.459478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.464209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.463525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.464209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.464068    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.464209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.464143    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.471388 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.470956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.471388 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.471002    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-config\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.471388 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.471042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.471388 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.471309    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.471705 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.471479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-web-config\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.471705 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.471640    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.473188 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.472937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78lp\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-kube-api-access-d78lp\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.473188 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.472956    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.473560 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.473536    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-config-out\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.474705 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.474015    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.474705 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.474615    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.483501 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.481688    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d4s8q"]
Jul 28 10:24:34.492531 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:34.492450    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081537ec_b941_46e9_b73e_f156ddb7585c.slice/crio-fbeda53387233b1cbab625c1af53f38936d112df11052f378849a576ad5d0acd WatchSource:0}: Error finding container fbeda53387233b1cbab625c1af53f38936d112df11052f378849a576ad5d0acd: Status 404 returned error can't find the container with id fbeda53387233b1cbab625c1af53f38936d112df11052f378849a576ad5d0acd
Jul 28 10:24:34.557240 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.556905    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:24:34.624813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.624772    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nsc54"]
Jul 28 10:24:34.649954 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.648493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d5g2m"]
Jul 28 10:24:34.654387 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:34.654345    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9311b88_8997_40d9_a0ea_5f5b5beca84f.slice/crio-b013410fa17637822f93c3d8cd72413abe8caa0b1ec2111e14792f76362f0da2 WatchSource:0}: Error finding container b013410fa17637822f93c3d8cd72413abe8caa0b1ec2111e14792f76362f0da2: Status 404 returned error can't find the container with id b013410fa17637822f93c3d8cd72413abe8caa0b1ec2111e14792f76362f0da2
Jul 28 10:24:34.763654 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:34.762536    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:24:34.767298 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:24:34.767239    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccef260d_4024_496b_99fb_44fd5d8e63f0.slice/crio-e46218a526a79748bb47fe3064eddba4e55a019fdfc705d30511a7682578607e WatchSource:0}: Error finding container e46218a526a79748bb47fe3064eddba4e55a019fdfc705d30511a7682578607e: Status 404 returned error can't find the container with id e46218a526a79748bb47fe3064eddba4e55a019fdfc705d30511a7682578607e
Jul 28 10:24:35.102297 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.102222    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"0aab8b3ad91d140ee3653c1e6a12530139892e67b092733fce429e8b19426faf"}
Jul 28 10:24:35.102758 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.102336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"37b1f66546e55926c9d31affbb9bb0a8452428f906d718c91f74316135f54b04"}
Jul 28 10:24:35.102758 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.102418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"14bb49ba5308680aeccabcaceff4e774c3af0687e7297edab140f1e0d9089ad3"}
Jul 28 10:24:35.102758 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.102461    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"2712aab1ff3ef7a6263a3e077715217e0fe621388169414a27bf27dbcac70b09"}
Jul 28 10:24:35.112341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.112224    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df" exitCode=0
Jul 28 10:24:35.112504 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.112375    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"}
Jul 28 10:24:35.112504 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.112494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"e46218a526a79748bb47fe3064eddba4e55a019fdfc705d30511a7682578607e"}
Jul 28 10:24:35.119985 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.119946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nsc54" event={"ID":"3d0ae63a-a197-421c-b2f6-f56f891ed32b","Type":"ContainerStarted","Data":"4af12599215f5a88d772630af12e04bec6c7282482ec8ea1affd083474fa3be7"}
Jul 28 10:24:35.127499 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.125058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d5g2m" event={"ID":"c9311b88-8997-40d9-a0ea-5f5b5beca84f","Type":"ContainerStarted","Data":"b013410fa17637822f93c3d8cd72413abe8caa0b1ec2111e14792f76362f0da2"}
Jul 28 10:24:35.128738 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:35.128635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d4s8q" event={"ID":"081537ec-b941-46e9-b73e-f156ddb7585c","Type":"ContainerStarted","Data":"fbeda53387233b1cbab625c1af53f38936d112df11052f378849a576ad5d0acd"}
Jul 28 10:24:51.215423 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.214318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-bb7db" event={"ID":"3bb5f1bb-5a0c-46da-a128-968330d00d73","Type":"ContainerStarted","Data":"486019efaa65f897dd97e2d6c656b2f0501d5bbb4eb9f098e88a3e310978591d"}
Jul 28 10:24:51.215423 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.214665    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:51.216644 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.216313    2578 patch_prober.go:28] interesting pod/downloads-7c97dc4555-bb7db container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body=
Jul 28 10:24:51.216644 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.216371    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-bb7db" podUID="3bb5f1bb-5a0c-46da-a128-968330d00d73" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused"
Jul 28 10:24:51.220128 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.220100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerStarted","Data":"f4b26b2b7a0adb1ebf3a4adeda028052b06648f38d40ec2e0a803964fb597915"}
Jul 28 10:24:51.224844 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.222685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw" event={"ID":"bf3517c1-cacd-4ee7-af11-65d55097e2d5","Type":"ContainerStarted","Data":"ca485eee43e3116401c5f130d80951bf7f8db2c5ef98fd510dc7a5d2e12c7f8d"}
Jul 28 10:24:51.224844 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.224254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:51.230301 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.230191    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"}
Jul 28 10:24:51.231537 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.231493    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw"
Jul 28 10:24:51.232086 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.231991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nsc54" event={"ID":"3d0ae63a-a197-421c-b2f6-f56f891ed32b","Type":"ContainerStarted","Data":"f632168d2e0707a5a353faa444aab309c6caa46c5a285fcbadc9ada5bb02d993"}
Jul 28 10:24:51.232086 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.232050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:24:51.234061 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.234040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d5g2m" event={"ID":"c9311b88-8997-40d9-a0ea-5f5b5beca84f","Type":"ContainerStarted","Data":"cad7edee756e9dd30b02d2580a94da36e624fc01e9dcd234fd669a08e13ed298"}
Jul 28 10:24:51.236981 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.236932    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-bb7db" podStartSLOduration=0.823276498 podStartE2EDuration="25.236918308s" totalImagesPullingTime="24.41364181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.235127129 +0000 UTC m=+82.277219282" watchObservedRunningTime="2026-07-28 10:24:51.236918308 +0000 UTC m=+82.279010464"
Jul 28 10:24:51.237896 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.237873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d4s8q" event={"ID":"081537ec-b941-46e9-b73e-f156ddb7585c","Type":"ContainerStarted","Data":"8dfe9182ff767aa6a23b73cc3cb8e1c18d81f2012a9a2d124037fad24b64f723"}
Jul 28 10:24:51.243426 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.243402    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845c87956d-n52dk" event={"ID":"474a6af3-842d-47aa-9360-fac0e6d87829","Type":"ContainerStarted","Data":"86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a"}
Jul 28 10:24:51.269376 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.268137    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.951729746 podStartE2EDuration="23.268115883s" totalImagesPullingTime="18.298036883s" totalInitContainerRuntime="18.349254ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.265960489 +0000 UTC m=+82.308052643" watchObservedRunningTime="2026-07-28 10:24:51.268115883 +0000 UTC m=+82.310208042"
Jul 28 10:24:51.283544 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.283239    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-jd9cw" podStartSLOduration=2.482113621 podStartE2EDuration="19.283218764s" totalImagesPullingTime="16.801105143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.280871565 +0000 UTC m=+82.322963719" watchObservedRunningTime="2026-07-28 10:24:51.283218764 +0000 UTC m=+82.325310917"
Jul 28 10:24:51.304603 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.304460    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d5g2m" podStartSLOduration=65.09885991 podStartE2EDuration="1m21.30443719s" totalImagesPullingTime="16.20557728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.302067554 +0000 UTC m=+82.344159708" watchObservedRunningTime="2026-07-28 10:24:51.30443719 +0000 UTC m=+82.346529343"
Jul 28 10:24:51.323364 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.323303    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nsc54" podStartSLOduration=66.062172114 podStartE2EDuration="1m22.323256597s" totalImagesPullingTime="16.261084483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:23:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.322103128 +0000 UTC m=+82.364195283" watchObservedRunningTime="2026-07-28 10:24:51.323256597 +0000 UTC m=+82.365348750"
Jul 28 10:24:51.349243 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.349187    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d4s8q" podStartSLOduration=64.946545969 podStartE2EDuration="1m21.349172755s" totalImagesPullingTime="16.402626786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:23:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.347085623 +0000 UTC m=+82.389177777" watchObservedRunningTime="2026-07-28 10:24:51.349172755 +0000 UTC m=+82.391264947"
Jul 28 10:24:51.368869 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:51.368811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-845c87956d-n52dk" podStartSLOduration=1.9253619560000002 podStartE2EDuration="20.368787432s" totalImagesPullingTime="18.443425476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:51.368173073 +0000 UTC m=+82.410265227" watchObservedRunningTime="2026-07-28 10:24:51.368787432 +0000 UTC m=+82.410879586"
Jul 28 10:24:52.259383 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:52.259320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"}
Jul 28 10:24:52.263536 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:52.263426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d5g2m" event={"ID":"c9311b88-8997-40d9-a0ea-5f5b5beca84f","Type":"ContainerStarted","Data":"3d09fe07934f9f78f63791cc042ac9141326fe5110e9cae1821f68d22486b83d"}
Jul 28 10:24:52.276219 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:52.276175    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-bb7db"
Jul 28 10:24:52.294517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:52.294430    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:52.294807 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:52.294751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:52.301614 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:52.301582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:53.275077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:53.275047    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:24:54.081731 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:54.081689    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845c87956d-n52dk"]
Jul 28 10:24:55.282790 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:55.282748    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"}
Jul 28 10:24:55.283264 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:55.282828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"}
Jul 28 10:24:55.283264 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:55.282867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"}
Jul 28 10:24:55.283264 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:55.282907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerStarted","Data":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"}
Jul 28 10:24:55.334936 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:55.334838    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.46915477 podStartE2EDuration="21.33481387s" totalImagesPullingTime="18.833408871s" totalInitContainerRuntime="32.250229ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:24:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:24:55.332828733 +0000 UTC m=+86.374920887" watchObservedRunningTime="2026-07-28 10:24:55.33481387 +0000 UTC m=+86.376906023"
Jul 28 10:24:59.558392 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:24:59.558346    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:20.304903 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.304849    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-845c87956d-n52dk" podUID="474a6af3-842d-47aa-9360-fac0e6d87829" containerName="console" containerID="cri-o://86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a" gracePeriod=15
Jul 28 10:25:20.545833 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.545807    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:25:20.692157 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692107    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-oauth-serving-cert\") pod \"474a6af3-842d-47aa-9360-fac0e6d87829\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") "
Jul 28 10:25:20.692383 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-service-ca\") pod \"474a6af3-842d-47aa-9360-fac0e6d87829\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") "
Jul 28 10:25:20.692383 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-oauth-config\") pod \"474a6af3-842d-47aa-9360-fac0e6d87829\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") "
Jul 28 10:25:20.692383 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-console-config\") pod \"474a6af3-842d-47aa-9360-fac0e6d87829\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") "
Jul 28 10:25:20.692383 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-serving-cert\") pod \"474a6af3-842d-47aa-9360-fac0e6d87829\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") "
Jul 28 10:25:20.692383 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sppcl\" (UniqueName: \"kubernetes.io/projected/474a6af3-842d-47aa-9360-fac0e6d87829-kube-api-access-sppcl\") pod \"474a6af3-842d-47aa-9360-fac0e6d87829\" (UID: \"474a6af3-842d-47aa-9360-fac0e6d87829\") "
Jul 28 10:25:20.692663 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "474a6af3-842d-47aa-9360-fac0e6d87829" (UID: "474a6af3-842d-47aa-9360-fac0e6d87829"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:20.692729 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-service-ca" (OuterVolumeSpecName: "service-ca") pod "474a6af3-842d-47aa-9360-fac0e6d87829" (UID: "474a6af3-842d-47aa-9360-fac0e6d87829"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:20.692777 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.692722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-console-config" (OuterVolumeSpecName: "console-config") pod "474a6af3-842d-47aa-9360-fac0e6d87829" (UID: "474a6af3-842d-47aa-9360-fac0e6d87829"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:20.694862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.694834    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474a6af3-842d-47aa-9360-fac0e6d87829-kube-api-access-sppcl" (OuterVolumeSpecName: "kube-api-access-sppcl") pod "474a6af3-842d-47aa-9360-fac0e6d87829" (UID: "474a6af3-842d-47aa-9360-fac0e6d87829"). InnerVolumeSpecName "kube-api-access-sppcl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:20.694862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.694839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "474a6af3-842d-47aa-9360-fac0e6d87829" (UID: "474a6af3-842d-47aa-9360-fac0e6d87829"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:20.694998 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.694901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "474a6af3-842d-47aa-9360-fac0e6d87829" (UID: "474a6af3-842d-47aa-9360-fac0e6d87829"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:20.793752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.793712    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sppcl\" (UniqueName: \"kubernetes.io/projected/474a6af3-842d-47aa-9360-fac0e6d87829-kube-api-access-sppcl\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:20.793752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.793744    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-oauth-serving-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:20.793752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.793754    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:20.793752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.793763    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-oauth-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:20.794005 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.793773    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/474a6af3-842d-47aa-9360-fac0e6d87829-console-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:20.794005 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:20.793781    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/474a6af3-842d-47aa-9360-fac0e6d87829-console-serving-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:21.390636 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.390602    2578 generic.go:356] "Generic (PLEG): container finished" podID="474a6af3-842d-47aa-9360-fac0e6d87829" containerID="86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a" exitCode=0
Jul 28 10:25:21.391099 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.390690    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845c87956d-n52dk" event={"ID":"474a6af3-842d-47aa-9360-fac0e6d87829","Type":"ContainerDied","Data":"86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a"}
Jul 28 10:25:21.391099 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.390700    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-845c87956d-n52dk"
Jul 28 10:25:21.391099 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.390758    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-845c87956d-n52dk" event={"ID":"474a6af3-842d-47aa-9360-fac0e6d87829","Type":"ContainerDied","Data":"bf855a67e924e03f7f2b9cd6e6f7d06f84b889d658c81560f29fb870c092bc4c"}
Jul 28 10:25:21.391099 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.390777    2578 scope.go:117] "RemoveContainer" containerID="86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a"
Jul 28 10:25:21.400448 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.400426    2578 scope.go:117] "RemoveContainer" containerID="86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a"
Jul 28 10:25:21.400793 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:21.400773    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a\": container with ID starting with 86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a not found: ID does not exist" containerID="86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a"
Jul 28 10:25:21.400837 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.400802    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a"} err="failed to get container status \"86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a\": rpc error: code = NotFound desc = could not find container \"86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a\": container with ID starting with 86bbfec08fe1977a0accfa61254a9df90885701fc97ffc0f1898ddd11b1e738a not found: ID does not exist"
Jul 28 10:25:21.413915 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.413882    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-845c87956d-n52dk"]
Jul 28 10:25:21.418202 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.418168    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-845c87956d-n52dk"]
Jul 28 10:25:21.494633 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:21.494599    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="474a6af3-842d-47aa-9360-fac0e6d87829" path="/var/lib/kubelet/pods/474a6af3-842d-47aa-9360-fac0e6d87829/volumes"
Jul 28 10:25:22.265974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:22.265938    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nsc54"
Jul 28 10:25:34.557974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:34.557932    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:34.636923 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:34.636894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:35.459318 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:35.459291    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:48.271470 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271427    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:25:48.271873 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271721    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="alertmanager" containerID="cri-o://50b0201fd67e40766be161fce31401fadc7f761b8e371fc8dfd2dae83c2c389a" gracePeriod=120
Jul 28 10:25:48.271873 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271784    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-metric" containerID="cri-o://0aab8b3ad91d140ee3653c1e6a12530139892e67b092733fce429e8b19426faf" gracePeriod=120
Jul 28 10:25:48.271873 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271843    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-web" containerID="cri-o://14bb49ba5308680aeccabcaceff4e774c3af0687e7297edab140f1e0d9089ad3" gracePeriod=120
Jul 28 10:25:48.272037 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271850    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy" containerID="cri-o://37b1f66546e55926c9d31affbb9bb0a8452428f906d718c91f74316135f54b04" gracePeriod=120
Jul 28 10:25:48.272037 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271872    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="config-reloader" containerID="cri-o://2712aab1ff3ef7a6263a3e077715217e0fe621388169414a27bf27dbcac70b09" gracePeriod=120
Jul 28 10:25:48.272037 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.271849    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="prom-label-proxy" containerID="cri-o://f4b26b2b7a0adb1ebf3a4adeda028052b06648f38d40ec2e0a803964fb597915" gracePeriod=120
Jul 28 10:25:48.495815 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495777    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="f4b26b2b7a0adb1ebf3a4adeda028052b06648f38d40ec2e0a803964fb597915" exitCode=0
Jul 28 10:25:48.495815 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495804    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="37b1f66546e55926c9d31affbb9bb0a8452428f906d718c91f74316135f54b04" exitCode=0
Jul 28 10:25:48.495815 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495813    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="2712aab1ff3ef7a6263a3e077715217e0fe621388169414a27bf27dbcac70b09" exitCode=0
Jul 28 10:25:48.495815 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495822    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="50b0201fd67e40766be161fce31401fadc7f761b8e371fc8dfd2dae83c2c389a" exitCode=0
Jul 28 10:25:48.496074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"f4b26b2b7a0adb1ebf3a4adeda028052b06648f38d40ec2e0a803964fb597915"}
Jul 28 10:25:48.496074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495907    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"37b1f66546e55926c9d31affbb9bb0a8452428f906d718c91f74316135f54b04"}
Jul 28 10:25:48.496074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"2712aab1ff3ef7a6263a3e077715217e0fe621388169414a27bf27dbcac70b09"}
Jul 28 10:25:48.496074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:48.495960    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"50b0201fd67e40766be161fce31401fadc7f761b8e371fc8dfd2dae83c2c389a"}
Jul 28 10:25:49.503429 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.503385    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="0aab8b3ad91d140ee3653c1e6a12530139892e67b092733fce429e8b19426faf" exitCode=0
Jul 28 10:25:49.503429 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.503413    2578 generic.go:356] "Generic (PLEG): container finished" podID="ffc5306a-0689-4813-a1a3-4bca52929958" containerID="14bb49ba5308680aeccabcaceff4e774c3af0687e7297edab140f1e0d9089ad3" exitCode=0
Jul 28 10:25:49.503874 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.503459    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"0aab8b3ad91d140ee3653c1e6a12530139892e67b092733fce429e8b19426faf"}
Jul 28 10:25:49.503874 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.503565    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"14bb49ba5308680aeccabcaceff4e774c3af0687e7297edab140f1e0d9089ad3"}
Jul 28 10:25:49.526091 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.526056    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:25:49.528995 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.528973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnw89\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-kube-api-access-qnw89\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529080 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529009    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-metrics-client-ca\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529080 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529080 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529057    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-config-volume\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529226 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529180    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-main-db\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529297 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529221    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-trusted-ca-bundle\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529297 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529265    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-main-tls\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529400 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529310    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-web-config\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529400 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529342    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-cluster-tls-config\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529400 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529375    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-tls-assets\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529411    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:49.529559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529449    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-web\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529712 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529556    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-config-out\") pod \"ffc5306a-0689-4813-a1a3-4bca52929958\" (UID: \"ffc5306a-0689-4813-a1a3-4bca52929958\") "
Jul 28 10:25:49.529712 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:49.529826 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529784    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-main-db\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.529826 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.529804    2578 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-metrics-client-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.530675 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.530630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:49.532598 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.532566    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-kube-api-access-qnw89" (OuterVolumeSpecName: "kube-api-access-qnw89") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "kube-api-access-qnw89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:49.532995 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.532967    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-config-volume" (OuterVolumeSpecName: "config-volume") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.533854 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.533824    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.533978 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.533954    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.534043 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.534021    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:49.534350 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.534315    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.534625 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.534579    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.534625 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.534599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-config-out" (OuterVolumeSpecName: "config-out") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:49.537730 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.537567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.545457 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.545417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-web-config" (OuterVolumeSpecName: "web-config") pod "ffc5306a-0689-4813-a1a3-4bca52929958" (UID: "ffc5306a-0689-4813-a1a3-4bca52929958"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630197    2578 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ffc5306a-0689-4813-a1a3-4bca52929958-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630231    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-main-tls\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630241    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-web-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630251    2578 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-cluster-tls-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630260    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-tls-assets\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630293    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630303    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630312    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ffc5306a-0689-4813-a1a3-4bca52929958-config-out\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630320    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnw89\" (UniqueName: \"kubernetes.io/projected/ffc5306a-0689-4813-a1a3-4bca52929958-kube-api-access-qnw89\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630699 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630328    2578 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:49.630699 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:49.630337    2578 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ffc5306a-0689-4813-a1a3-4bca52929958-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:50.510729 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.510697    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ffc5306a-0689-4813-a1a3-4bca52929958","Type":"ContainerDied","Data":"3e709d930e29b43201aa39e3d444382efdb454249d94aeca645a9b6349a479c9"}
Jul 28 10:25:50.510729 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.510713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 10:25:50.511209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.510774    2578 scope.go:117] "RemoveContainer" containerID="f4b26b2b7a0adb1ebf3a4adeda028052b06648f38d40ec2e0a803964fb597915"
Jul 28 10:25:50.519287 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.519240    2578 scope.go:117] "RemoveContainer" containerID="0aab8b3ad91d140ee3653c1e6a12530139892e67b092733fce429e8b19426faf"
Jul 28 10:25:50.526247 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.526231    2578 scope.go:117] "RemoveContainer" containerID="37b1f66546e55926c9d31affbb9bb0a8452428f906d718c91f74316135f54b04"
Jul 28 10:25:50.532852 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.532833    2578 scope.go:117] "RemoveContainer" containerID="14bb49ba5308680aeccabcaceff4e774c3af0687e7297edab140f1e0d9089ad3"
Jul 28 10:25:50.535432 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.535409    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:25:50.539413 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.539389    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 10:25:50.540694 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.540680    2578 scope.go:117] "RemoveContainer" containerID="2712aab1ff3ef7a6263a3e077715217e0fe621388169414a27bf27dbcac70b09"
Jul 28 10:25:50.547746 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.547728    2578 scope.go:117] "RemoveContainer" containerID="50b0201fd67e40766be161fce31401fadc7f761b8e371fc8dfd2dae83c2c389a"
Jul 28 10:25:50.554893 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:50.554875    2578 scope.go:117] "RemoveContainer" containerID="38341bebbaf3a7092245c4fbad458f65f8e1415d6eff052ba6564121d8c59591"
Jul 28 10:25:51.492982 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:51.492940    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" path="/var/lib/kubelet/pods/ffc5306a-0689-4813-a1a3-4bca52929958/volumes"
Jul 28 10:25:52.499779 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.499740    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:25:52.500214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.500056    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="prometheus" containerID="cri-o://90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2" gracePeriod=600
Jul 28 10:25:52.500214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.500066    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy" containerID="cri-o://9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a" gracePeriod=600
Jul 28 10:25:52.500214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.500128    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="config-reloader" containerID="cri-o://71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8" gracePeriod=600
Jul 28 10:25:52.500214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.500123    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="thanos-sidecar" containerID="cri-o://ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7" gracePeriod=600
Jul 28 10:25:52.500214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.500158    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-web" containerID="cri-o://04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6" gracePeriod=600
Jul 28 10:25:52.500214 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.500133    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b" gracePeriod=600
Jul 28 10:25:52.766493 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.766464    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:52.855124 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d78lp\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-kube-api-access-d78lp\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-config-out\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855171    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-metrics-client-ca\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855199    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-serving-certs-ca-bundle\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855341 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855324    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-db\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855378    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-thanos-prometheus-http-client-file\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855406    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-config\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855432    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-1\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855459    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855492    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-web-config\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855525    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-tls\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855559 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-kubelet-serving-ca-bundle\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-trusted-ca-bundle\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-kube-rbac-proxy\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-0\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-tls-assets\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855746    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-metrics-client-certs\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855778    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-grpc-tls\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-2\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855849    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ccef260d-4024-496b-99fb-44fd5d8e63f0\" (UID: \"ccef260d-4024-496b-99fb-44fd5d8e63f0\") "
Jul 28 10:25:52.855895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.855897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.856752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.856064    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.856752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.856083    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.856752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.856098    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.856752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.856112    2578 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-configmap-metrics-client-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.856752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.856451    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.856752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.856592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:52.857943 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.857681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.858372 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.858083    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-config-out" (OuterVolumeSpecName: "config-out") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:25:52.859082 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.859055    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:25:52.859284 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.859245    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.861029 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861002    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.861344 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861250    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.861473 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861247    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-kube-api-access-d78lp" (OuterVolumeSpecName: "kube-api-access-d78lp") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "kube-api-access-d78lp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:52.861567 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.861678 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:25:52.861678 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861627    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.861964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861677    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.861964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.861773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.862500 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.862480    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-config" (OuterVolumeSpecName: "config") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.869743 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.869710    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-web-config" (OuterVolumeSpecName: "web-config") pod "ccef260d-4024-496b-99fb-44fd5d8e63f0" (UID: "ccef260d-4024-496b-99fb-44fd5d8e63f0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:25:52.957241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957194    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78lp\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-kube-api-access-d78lp\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957231    2578 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-config-out\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957241    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-db\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957250    2578 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957261    2578 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957295    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957304    2578 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-web-config\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957312    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957320    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957328    2578 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-kube-rbac-proxy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957337    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957345    2578 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ccef260d-4024-496b-99fb-44fd5d8e63f0-tls-assets\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957353    2578 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-metrics-client-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957361    2578 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-grpc-tls\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957369    2578 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ccef260d-4024-496b-99fb-44fd5d8e63f0-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:52.957533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:52.957376    2578 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ccef260d-4024-496b-99fb-44fd5d8e63f0-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:25:53.529256 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529224    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b" exitCode=0
Jul 28 10:25:53.529256 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529251    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a" exitCode=0
Jul 28 10:25:53.529256 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529259    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6" exitCode=0
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529285    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7" exitCode=0
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529298    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8" exitCode=0
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529306    2578 generic.go:356] "Generic (PLEG): container finished" podID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2" exitCode=0
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529325    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"}
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"}
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529431    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"}
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"}
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529466    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"}
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"}
Jul 28 10:25:53.529737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.529537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ccef260d-4024-496b-99fb-44fd5d8e63f0","Type":"ContainerDied","Data":"e46218a526a79748bb47fe3064eddba4e55a019fdfc705d30511a7682578607e"}
Jul 28 10:25:53.537484 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.537455    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.544569 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.544551    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.551490 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.551461    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:25:53.552138 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.552121    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.555494 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.555472    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:25:53.559657 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.559629    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.566679 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.566661    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.574232 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.574210    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.581616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.581514    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.581964 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.581925    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.582062 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.581979    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"} err="failed to get container status \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist"
Jul 28 10:25:53.582062 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582006    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.582375 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.582354    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.582460 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582379    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"} err="failed to get container status \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist"
Jul 28 10:25:53.582460 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582394    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.582698 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.582674    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.582749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582705    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"} err="failed to get container status \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist"
Jul 28 10:25:53.582749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582722    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.582873 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582853    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:25:53.582989 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.582975    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.583027 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.582993    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"} err="failed to get container status \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist"
Jul 28 10:25:53.583027 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583005    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.583146 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583129    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-thanos"
Jul 28 10:25:53.583197 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-thanos"
Jul 28 10:25:53.583197 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583164    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="474a6af3-842d-47aa-9360-fac0e6d87829" containerName="console"
Jul 28 10:25:53.583197 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="474a6af3-842d-47aa-9360-fac0e6d87829" containerName="console"
Jul 28 10:25:53.583197 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583185    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="init-config-reloader"
Jul 28 10:25:53.583197 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583191    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="init-config-reloader"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583202    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="config-reloader"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583207    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="config-reloader"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583217    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="thanos-sidecar"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583222    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="thanos-sidecar"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583230    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-web"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583235    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-web"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583243    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="prom-label-proxy"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583248    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="prom-label-proxy"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583254    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.583245    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583301    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"} err="failed to get container status \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583324    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583259    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583365    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-web"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583370    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-web"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583376    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583381    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583389    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="prometheus"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583393    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="prometheus"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583401    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="alertmanager"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583406    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="alertmanager"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583411    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-metric"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583416    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-metric"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583421    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="init-config-reloader"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583426    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="init-config-reloader"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583432    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="config-reloader"
Jul 28 10:25:53.583444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583440    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="config-reloader"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583493    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="alertmanager"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583502    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="thanos-sidecar"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583513    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583519    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-metric"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583525    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-web"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583530    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="prometheus"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583536    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="kube-rbac-proxy-web"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583541    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="prom-label-proxy"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583546    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="474a6af3-842d-47aa-9360-fac0e6d87829" containerName="console"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583550    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffc5306a-0689-4813-a1a3-4bca52929958" containerName="config-reloader"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583555    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="config-reloader"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583560    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy-thanos"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583568    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" containerName="kube-rbac-proxy"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.583585    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583601    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"} err="failed to get container status \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583615    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:25:53.583804    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583821    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"} err="failed to get container status \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.583833    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584007    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"} err="failed to get container status \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584018    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584230    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"} err="failed to get container status \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist"
Jul 28 10:25:53.584403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584248    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584509    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"} err="failed to get container status \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584523    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584726    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"} err="failed to get container status \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584738    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584937    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"} err="failed to get container status \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.584950    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585146    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"} err="failed to get container status \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist"
Jul 28 10:25:53.585159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585158    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.585691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585437    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"} err="failed to get container status \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist"
Jul 28 10:25:53.585691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585459    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.585763 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585704    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"} err="failed to get container status \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist"
Jul 28 10:25:53.585763 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585726    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.586011 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.585987    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"} err="failed to get container status \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist"
Jul 28 10:25:53.586011 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586011    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.586254 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586233    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"} err="failed to get container status \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist"
Jul 28 10:25:53.586325 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586256    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.586521 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586500    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"} err="failed to get container status \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist"
Jul 28 10:25:53.586521 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586519    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.586867 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586840    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"} err="failed to get container status \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist"
Jul 28 10:25:53.586951 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.586875    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.587129 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587109    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"} err="failed to get container status \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist"
Jul 28 10:25:53.587193 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587130    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.587406 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587384    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"} err="failed to get container status \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist"
Jul 28 10:25:53.587471 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587409    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.587471 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587396    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.587904 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587647    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"} err="failed to get container status \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist"
Jul 28 10:25:53.587904 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.587671    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.588109 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588088    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"} err="failed to get container status \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist"
Jul 28 10:25:53.588182 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588110    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.588331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"} err="failed to get container status \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist"
Jul 28 10:25:53.588380 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588332    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.588484 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588466    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"} err="failed to get container status \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist"
Jul 28 10:25:53.588537 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588484    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.588654 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588639    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"} err="failed to get container status \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist"
Jul 28 10:25:53.588699 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588653    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.588846 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588829    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"} err="failed to get container status \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist"
Jul 28 10:25:53.588907 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.588848    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.589039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589023    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"} err="failed to get container status \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist"
Jul 28 10:25:53.589104 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589041    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.589310 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589257    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"} err="failed to get container status \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist"
Jul 28 10:25:53.589421 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589313    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.589547 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589526    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"} err="failed to get container status \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist"
Jul 28 10:25:53.589597 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589548    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.589813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589795    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"} err="failed to get container status \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist"
Jul 28 10:25:53.589856 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.589813    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.590088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590065    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"} err="failed to get container status \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist"
Jul 28 10:25:53.590209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590090    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.590209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 10:25:53.590209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590113    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 10:25:53.590209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590115    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 10:25:53.590432 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590338    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"} err="failed to get container status \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist"
Jul 28 10:25:53.590432 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590359    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.590792 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590528    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 10:25:53.590792 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590602    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 10:25:53.590792 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590667    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 10:25:53.590792 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 10:25:53.590792 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590742    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"} err="failed to get container status \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist"
Jul 28 10:25:53.590792 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590772    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.591131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590759    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 10:25:53.591131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590837    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 10:25:53.591131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 10:25:53.591131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.590926    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 10:25:53.591131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.591108    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 10:25:53.591385 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.591185    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-ao6el0591c0sv\""
Jul 28 10:25:53.591385 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.591211    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lfjmt\""
Jul 28 10:25:53.591493 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.591478    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 10:25:53.592160 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.592131    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"} err="failed to get container status \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist"
Jul 28 10:25:53.592252 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.592163    2578 scope.go:117] "RemoveContainer" containerID="ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"
Jul 28 10:25:53.592507 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.592480    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b"} err="failed to get container status \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": rpc error: code = NotFound desc = could not find container \"ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b\": container with ID starting with ab2824cc1facdf74966f010b6662e5f23b8775742241a2b4da1588527ce0b63b not found: ID does not exist"
Jul 28 10:25:53.592574 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.592510    2578 scope.go:117] "RemoveContainer" containerID="9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"
Jul 28 10:25:53.592784 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.592758    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a"} err="failed to get container status \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": rpc error: code = NotFound desc = could not find container \"9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a\": container with ID starting with 9446aa64100c1971c12fcc013baeb30411e5de36f873e6335769e960e329a50a not found: ID does not exist"
Jul 28 10:25:53.592864 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.592787    2578 scope.go:117] "RemoveContainer" containerID="04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"
Jul 28 10:25:53.593045 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593021    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6"} err="failed to get container status \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": rpc error: code = NotFound desc = could not find container \"04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6\": container with ID starting with 04dd8e8578ac2ae4d68fed4a528b90cb097dc77654dd1696e6d0ed9f88c432c6 not found: ID does not exist"
Jul 28 10:25:53.593109 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593048    2578 scope.go:117] "RemoveContainer" containerID="ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"
Jul 28 10:25:53.593333 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593312    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7"} err="failed to get container status \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": rpc error: code = NotFound desc = could not find container \"ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7\": container with ID starting with ab0f63d488ec82fb518f9b33e7660bcf50e8205b254d4cebf9638e35df2a0cb7 not found: ID does not exist"
Jul 28 10:25:53.593450 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593334    2578 scope.go:117] "RemoveContainer" containerID="71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"
Jul 28 10:25:53.593607 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593585    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8"} err="failed to get container status \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": rpc error: code = NotFound desc = could not find container \"71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8\": container with ID starting with 71a29574540ec582c9f3ba0c2b7148b93c0a49955b09e44bf74c1ac4f69ba5d8 not found: ID does not exist"
Jul 28 10:25:53.593729 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593609    2578 scope.go:117] "RemoveContainer" containerID="90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"
Jul 28 10:25:53.593951 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2"} err="failed to get container status \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": rpc error: code = NotFound desc = could not find container \"90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2\": container with ID starting with 90e7d25d68e74a4b235b602c0391e10183d8eb51be4d6974ec75d1ea60b611d2 not found: ID does not exist"
Jul 28 10:25:53.593951 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.593910    2578 scope.go:117] "RemoveContainer" containerID="d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"
Jul 28 10:25:53.594189 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.594160    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df"} err="failed to get container status \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": rpc error: code = NotFound desc = could not find container \"d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df\": container with ID starting with d9ef95fd7a42d2ee2a7e365bf3bf6acab255f08e72da51d1e4ab5e129b4042df not found: ID does not exist"
Jul 28 10:25:53.594507 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.594487    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 10:25:53.596330 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.596310    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 10:25:53.604410 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.604373    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:25:53.661028 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.660983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661028 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661040    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-config-out\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661078    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661099    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661341    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661416    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-config\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661430    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldv9v\" (UniqueName: \"kubernetes.io/projected/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-kube-api-access-ldv9v\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661452    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661767 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661472    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-web-config\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661767 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.661767 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.661590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.762647 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.762647 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762580    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.762647 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.762647 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-config-out\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.762647 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.762647 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762822    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762864    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762905    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762931    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-config\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldv9v\" (UniqueName: \"kubernetes.io/projected/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-kube-api-access-ldv9v\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.762981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.763003    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-web-config\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.763070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.763102    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.763157    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.763445    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.763813 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.763661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.764793 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.764307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.764793 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.764740    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.764971 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.764839    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.765919 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.765894    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.766993 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.766896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.767558 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.767198    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.767558 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.767202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.767558 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.767486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.768187 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.767824    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.768187 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.767958    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-config-out\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.768319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.768221    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-config\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.768381 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.768325    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.768830 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.768804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.769177 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.769154    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.769237 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.769218    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.769315 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.769250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-web-config\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.771885 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.771864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldv9v\" (UniqueName: \"kubernetes.io/projected/c42d0cd7-b26b-4ef6-bfcc-d5325387f140-kube-api-access-ldv9v\") pod \"prometheus-k8s-0\" (UID: \"c42d0cd7-b26b-4ef6-bfcc-d5325387f140\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:53.897917 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:53.897878    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:25:54.039666 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:54.039592    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 10:25:54.042672 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:25:54.042642    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42d0cd7_b26b_4ef6_bfcc_d5325387f140.slice/crio-defecfab669480e899747ab6fa0de2d39b8d1364e3b5c378e3e86b849f189d78 WatchSource:0}: Error finding container defecfab669480e899747ab6fa0de2d39b8d1364e3b5c378e3e86b849f189d78: Status 404 returned error can't find the container with id defecfab669480e899747ab6fa0de2d39b8d1364e3b5c378e3e86b849f189d78
Jul 28 10:25:54.534284 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:54.534246    2578 generic.go:356] "Generic (PLEG): container finished" podID="c42d0cd7-b26b-4ef6-bfcc-d5325387f140" containerID="5cfd2b284b54a96be5ce97a30bad4a665c1aa47fc1f7aa5fed7b762093046fb4" exitCode=0
Jul 28 10:25:54.534749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:54.534343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerDied","Data":"5cfd2b284b54a96be5ce97a30bad4a665c1aa47fc1f7aa5fed7b762093046fb4"}
Jul 28 10:25:54.534749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:54.534413    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"defecfab669480e899747ab6fa0de2d39b8d1364e3b5c378e3e86b849f189d78"}
Jul 28 10:25:55.494131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.494096    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ccef260d-4024-496b-99fb-44fd5d8e63f0" path="/var/lib/kubelet/pods/ccef260d-4024-496b-99fb-44fd5d8e63f0/volumes"
Jul 28 10:25:55.543039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.543004    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"b9937f87e06b041041a3d12f260b61aac49b03e39e08d1c905390ac599a4242a"}
Jul 28 10:25:55.543443 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.543080    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"e35fa320e226d529a9eef2f7c01df1c78d7e8d76780fb51b8072612ee10cf04b"}
Jul 28 10:25:55.543443 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.543110    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"7d14aabe20dea412c9153c59ab313ccc3c196986a6711cd4c8d0bfdea33902ab"}
Jul 28 10:25:55.543443 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.543138    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"0e9b0980e52f599dcc551f76f159065f90360d5e26c55b174207355544f71ec6"}
Jul 28 10:25:55.543443 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.543163    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"d5d462109ba8bd505c554b89bd10ddc5499fa9c95b6eda2c3242f959165d0f34"}
Jul 28 10:25:55.543443 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.543189    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c42d0cd7-b26b-4ef6-bfcc-d5325387f140","Type":"ContainerStarted","Data":"1dcfb37077a8f1c2cab6862c387e0da4c91cb22c468277d6878d3ad1cf7f0048"}
Jul 28 10:25:55.575929 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:55.575870    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5562899850000003 podStartE2EDuration="2.575852389s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.562404ms" isStatefulPod=false podCreationTimestamp="2026-07-28 10:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:25:55.574784404 +0000 UTC m=+146.616876560" watchObservedRunningTime="2026-07-28 10:25:55.575852389 +0000 UTC m=+146.617944543"
Jul 28 10:25:58.898489 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:25:58.898443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:26:53.898233 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:26:53.898200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:26:53.914850 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:26:53.914822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:26:54.767925 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:26:54.767896    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 10:28:29.322140 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:28:29.322112    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:28:29.327093 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:28:29.327068    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:28:29.328535 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:28:29.328517    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 10:33:29.344210 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:33:29.344184    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:33:29.347463 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:33:29.347437    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:34:36.654743 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.654708    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"]
Jul 28 10:34:36.657916 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.657901    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.660635 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.660614    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 10:34:36.660770 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.660701    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-smsqv\""
Jul 28 10:34:36.661631 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.661611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 10:34:36.665791 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.665767    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"]
Jul 28 10:34:36.794189 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.794139    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9shjg\" (UniqueName: \"kubernetes.io/projected/a11d2c2d-5149-41d7-83b5-407655e5472a-kube-api-access-9shjg\") pod \"cert-manager-cainjector-559cdd8c87-gf7ph\" (UID: \"a11d2c2d-5149-41d7-83b5-407655e5472a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.794189 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.794193    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a11d2c2d-5149-41d7-83b5-407655e5472a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gf7ph\" (UID: \"a11d2c2d-5149-41d7-83b5-407655e5472a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.895184 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.895151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9shjg\" (UniqueName: \"kubernetes.io/projected/a11d2c2d-5149-41d7-83b5-407655e5472a-kube-api-access-9shjg\") pod \"cert-manager-cainjector-559cdd8c87-gf7ph\" (UID: \"a11d2c2d-5149-41d7-83b5-407655e5472a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.895184 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.895185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a11d2c2d-5149-41d7-83b5-407655e5472a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gf7ph\" (UID: \"a11d2c2d-5149-41d7-83b5-407655e5472a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.904125 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.904105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a11d2c2d-5149-41d7-83b5-407655e5472a-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-gf7ph\" (UID: \"a11d2c2d-5149-41d7-83b5-407655e5472a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.904358 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.904337    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9shjg\" (UniqueName: \"kubernetes.io/projected/a11d2c2d-5149-41d7-83b5-407655e5472a-kube-api-access-9shjg\") pod \"cert-manager-cainjector-559cdd8c87-gf7ph\" (UID: \"a11d2c2d-5149-41d7-83b5-407655e5472a\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:36.985129 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:36.985095    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"
Jul 28 10:34:37.111709 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:37.111683    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph"]
Jul 28 10:34:37.114167 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:34:37.114143    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda11d2c2d_5149_41d7_83b5_407655e5472a.slice/crio-7cb2ff90198cf14c5ac5242c965dba6c94675fb0d966f568b8f345b5f006aa4d WatchSource:0}: Error finding container 7cb2ff90198cf14c5ac5242c965dba6c94675fb0d966f568b8f345b5f006aa4d: Status 404 returned error can't find the container with id 7cb2ff90198cf14c5ac5242c965dba6c94675fb0d966f568b8f345b5f006aa4d
Jul 28 10:34:37.116151 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:37.116137    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:34:37.355986 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:37.355896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph" event={"ID":"a11d2c2d-5149-41d7-83b5-407655e5472a","Type":"ContainerStarted","Data":"7cb2ff90198cf14c5ac5242c965dba6c94675fb0d966f568b8f345b5f006aa4d"}
Jul 28 10:34:40.369328 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:40.369213    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph" event={"ID":"a11d2c2d-5149-41d7-83b5-407655e5472a","Type":"ContainerStarted","Data":"b66524b5e9810bb93152c974a508f44446fbf86d28a6e2cc24c83db325be52a6"}
Jul 28 10:34:40.391615 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:34:40.391566    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-gf7ph" podStartSLOduration=1.476984077 podStartE2EDuration="4.391549669s" totalImagesPullingTime="2.914565592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:34:40.389587246 +0000 UTC m=+671.431679399" watchObservedRunningTime="2026-07-28 10:34:40.391549669 +0000 UTC m=+671.433641821"
Jul 28 10:35:07.664429 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.664394    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"]
Jul 28 10:35:07.667701 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.667680    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.673790 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.673446    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 10:35:07.674058 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.674036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 10:35:07.674145 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.674109    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 10:35:07.674465 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.674450    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k7g2n\""
Jul 28 10:35:07.674761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.674745    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 10:35:07.685181 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.685153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"]
Jul 28 10:35:07.737036 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.736997    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe736835-a59d-4b26-ab49-1c42127c5344-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.737036 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.737037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe736835-a59d-4b26-ab49-1c42127c5344-tmp\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.737309 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.737063    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmjm\" (UniqueName: \"kubernetes.io/projected/fe736835-a59d-4b26-ab49-1c42127c5344-kube-api-access-chmjm\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.737309 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.737095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe736835-a59d-4b26-ab49-1c42127c5344-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.737309 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.737161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe736835-a59d-4b26-ab49-1c42127c5344-manifests\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.838033 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.837993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe736835-a59d-4b26-ab49-1c42127c5344-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.838033 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.838034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe736835-a59d-4b26-ab49-1c42127c5344-tmp\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.838336 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.838060    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmjm\" (UniqueName: \"kubernetes.io/projected/fe736835-a59d-4b26-ab49-1c42127c5344-kube-api-access-chmjm\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.838336 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.838169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe736835-a59d-4b26-ab49-1c42127c5344-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.838336 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.838216    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe736835-a59d-4b26-ab49-1c42127c5344-manifests\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.838555 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.838526    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/fe736835-a59d-4b26-ab49-1c42127c5344-manifests\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.840680 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.840644    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fe736835-a59d-4b26-ab49-1c42127c5344-tmp\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.840822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.840717    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fe736835-a59d-4b26-ab49-1c42127c5344-webhook-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.840822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.840787    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/fe736835-a59d-4b26-ab49-1c42127c5344-apiservice-cert\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.852376 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.852345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmjm\" (UniqueName: \"kubernetes.io/projected/fe736835-a59d-4b26-ab49-1c42127c5344-kube-api-access-chmjm\") pod \"opendatahub-operator-controller-manager-559f66f7dc-xxd76\" (UID: \"fe736835-a59d-4b26-ab49-1c42127c5344\") " pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:07.977657 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:07.977608    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:08.126716 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:08.126671    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"]
Jul 28 10:35:08.128943 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:35:08.128918    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe736835_a59d_4b26_ab49_1c42127c5344.slice/crio-0a2c3c22a68fbf4993eb34be78bfeb1a51c6c124b7ad6d5bf94a35aa1ff72283 WatchSource:0}: Error finding container 0a2c3c22a68fbf4993eb34be78bfeb1a51c6c124b7ad6d5bf94a35aa1ff72283: Status 404 returned error can't find the container with id 0a2c3c22a68fbf4993eb34be78bfeb1a51c6c124b7ad6d5bf94a35aa1ff72283
Jul 28 10:35:08.465949 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:08.465911    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76" event={"ID":"fe736835-a59d-4b26-ab49-1c42127c5344","Type":"ContainerStarted","Data":"0a2c3c22a68fbf4993eb34be78bfeb1a51c6c124b7ad6d5bf94a35aa1ff72283"}
Jul 28 10:35:11.480036 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:11.480000    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76" event={"ID":"fe736835-a59d-4b26-ab49-1c42127c5344","Type":"ContainerStarted","Data":"9c9f569fe106e295a1b9afcb3913fdc91a6ba55cc2164ec70c9355c327089ad0"}
Jul 28 10:35:12.485216 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:12.485121    2578 generic.go:356] "Generic (PLEG): container finished" podID="fe736835-a59d-4b26-ab49-1c42127c5344" containerID="9c9f569fe106e295a1b9afcb3913fdc91a6ba55cc2164ec70c9355c327089ad0" exitCode=0
Jul 28 10:35:12.485216 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:12.485187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76" event={"ID":"fe736835-a59d-4b26-ab49-1c42127c5344","Type":"ContainerDied","Data":"9c9f569fe106e295a1b9afcb3913fdc91a6ba55cc2164ec70c9355c327089ad0"}
Jul 28 10:35:13.492863 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:13.492828    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76" event={"ID":"fe736835-a59d-4b26-ab49-1c42127c5344","Type":"ContainerStarted","Data":"5d3865af9c54f893837eb4562f75d95e1b110000a37af8ff27f892febb011471"}
Jul 28 10:35:13.493252 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:13.492884    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:13.515650 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:13.515602    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76" podStartSLOduration=2.175135292 podStartE2EDuration="6.515585847s" totalImagesPullingTime="3.308331662s" totalInitContainerRuntime="1.032118893s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:13.514317454 +0000 UTC m=+704.556409624" watchObservedRunningTime="2026-07-28 10:35:13.515585847 +0000 UTC m=+704.557678049"
Jul 28 10:35:24.497849 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:24.497768    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-559f66f7dc-xxd76"
Jul 28 10:35:25.765757 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.765720    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"]
Jul 28 10:35:25.768887 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.768866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.771910 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.771871    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 10:35:25.772063 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.771878    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 10:35:25.773254 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.773235    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 10:35:25.773389 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.773283    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:35:25.773389 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.773261    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zr6t4\""
Jul 28 10:35:25.773389 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.773333    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 10:35:25.781243 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.781216    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"]
Jul 28 10:35:25.873002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.872963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9feac2b4-c761-441f-bf84-f8af780e6e44-metrics-cert\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.873002 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.873003    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmmv\" (UniqueName: \"kubernetes.io/projected/9feac2b4-c761-441f-bf84-f8af780e6e44-kube-api-access-zgmmv\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.873221 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.873027    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9feac2b4-c761-441f-bf84-f8af780e6e44-cert\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.873221 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.873137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9feac2b4-c761-441f-bf84-f8af780e6e44-manager-config\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.973770 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.973732    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9feac2b4-c761-441f-bf84-f8af780e6e44-metrics-cert\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.973770 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.973772    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmmv\" (UniqueName: \"kubernetes.io/projected/9feac2b4-c761-441f-bf84-f8af780e6e44-kube-api-access-zgmmv\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.974017 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.973793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9feac2b4-c761-441f-bf84-f8af780e6e44-cert\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.974017 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.973826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9feac2b4-c761-441f-bf84-f8af780e6e44-manager-config\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.974565 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.974543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9feac2b4-c761-441f-bf84-f8af780e6e44-manager-config\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.976708 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.976684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9feac2b4-c761-441f-bf84-f8af780e6e44-metrics-cert\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:25.977155 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:25.977128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9feac2b4-c761-441f-bf84-f8af780e6e44-cert\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:26.015234 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:26.015199    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmmv\" (UniqueName: \"kubernetes.io/projected/9feac2b4-c761-441f-bf84-f8af780e6e44-kube-api-access-zgmmv\") pod \"lws-controller-manager-7d8dfbcf79-56k9b\" (UID: \"9feac2b4-c761-441f-bf84-f8af780e6e44\") " pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:26.079322 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:26.079189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:26.245990 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:26.245958    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"]
Jul 28 10:35:26.276411 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:35:26.276379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9feac2b4_c761_441f_bf84_f8af780e6e44.slice/crio-e696cf4b3dba43dfb6c72bee41cbc5d29f01f3f80352e92f42be276b53593422 WatchSource:0}: Error finding container e696cf4b3dba43dfb6c72bee41cbc5d29f01f3f80352e92f42be276b53593422: Status 404 returned error can't find the container with id e696cf4b3dba43dfb6c72bee41cbc5d29f01f3f80352e92f42be276b53593422
Jul 28 10:35:26.548528 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:26.548490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b" event={"ID":"9feac2b4-c761-441f-bf84-f8af780e6e44","Type":"ContainerStarted","Data":"e696cf4b3dba43dfb6c72bee41cbc5d29f01f3f80352e92f42be276b53593422"}
Jul 28 10:35:29.566006 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:29.565961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b" event={"ID":"9feac2b4-c761-441f-bf84-f8af780e6e44","Type":"ContainerStarted","Data":"3fcf061be62e598114db447fbc47441436c5631500ef881e07b80042210ca22b"}
Jul 28 10:35:29.566512 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:29.566100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:29.588992 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:29.588933    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b" podStartSLOduration=2.243848004 podStartE2EDuration="4.588915274s" totalImagesPullingTime="2.34506727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:29.587702732 +0000 UTC m=+720.629794887" watchObservedRunningTime="2026-07-28 10:35:29.588915274 +0000 UTC m=+720.631007426"
Jul 28 10:35:37.277617 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.277580    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"]
Jul 28 10:35:37.282373 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.282348    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.285663 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.285645    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 10:35:37.286101 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.286082    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 10:35:37.286101 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.286099    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 10:35:37.286390 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.286376    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 10:35:37.286874 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.286857    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ntp8r\""
Jul 28 10:35:37.296840 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.296815    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"]
Jul 28 10:35:37.363646 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.363605    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsrxf\" (UniqueName: \"kubernetes.io/projected/41e507ed-b68a-40d0-bf1c-fa97e2005970-kube-api-access-dsrxf\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.363646 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.363652    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e507ed-b68a-40d0-bf1c-fa97e2005970-tmp\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.363896 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.363762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e507ed-b68a-40d0-bf1c-fa97e2005970-tls-certs\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.464518 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.464471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsrxf\" (UniqueName: \"kubernetes.io/projected/41e507ed-b68a-40d0-bf1c-fa97e2005970-kube-api-access-dsrxf\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.464518 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.464517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e507ed-b68a-40d0-bf1c-fa97e2005970-tmp\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.464782 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.464558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e507ed-b68a-40d0-bf1c-fa97e2005970-tls-certs\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.466963 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.466936    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/41e507ed-b68a-40d0-bf1c-fa97e2005970-tmp\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.467252 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.467232    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41e507ed-b68a-40d0-bf1c-fa97e2005970-tls-certs\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.481228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.481201    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsrxf\" (UniqueName: \"kubernetes.io/projected/41e507ed-b68a-40d0-bf1c-fa97e2005970-kube-api-access-dsrxf\") pod \"kube-auth-proxy-5d6b86846-5s4d2\" (UID: \"41e507ed-b68a-40d0-bf1c-fa97e2005970\") " pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.591896 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.591802    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"
Jul 28 10:35:37.766842 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:37.766734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2"]
Jul 28 10:35:37.769067 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:35:37.769043    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41e507ed_b68a_40d0_bf1c_fa97e2005970.slice/crio-b8f8ed90f0c7786ea2e98ac54b632c7fc46d251d0d8e5a1c22f0518aa9a840e3 WatchSource:0}: Error finding container b8f8ed90f0c7786ea2e98ac54b632c7fc46d251d0d8e5a1c22f0518aa9a840e3: Status 404 returned error can't find the container with id b8f8ed90f0c7786ea2e98ac54b632c7fc46d251d0d8e5a1c22f0518aa9a840e3
Jul 28 10:35:38.600747 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:38.600712    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2" event={"ID":"41e507ed-b68a-40d0-bf1c-fa97e2005970","Type":"ContainerStarted","Data":"b8f8ed90f0c7786ea2e98ac54b632c7fc46d251d0d8e5a1c22f0518aa9a840e3"}
Jul 28 10:35:40.572461 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:40.572439    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7d8dfbcf79-56k9b"
Jul 28 10:35:41.622407 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:41.622369    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2" event={"ID":"41e507ed-b68a-40d0-bf1c-fa97e2005970","Type":"ContainerStarted","Data":"a571a58dea6f446b9b090982dde377d08bb51cebf7d798309a14c383d292cd76"}
Jul 28 10:35:41.648123 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:35:41.648066    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d6b86846-5s4d2" podStartSLOduration=1.865638992 podStartE2EDuration="4.648050738s" totalImagesPullingTime="2.782411746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:35:41.646754169 +0000 UTC m=+732.688846313" watchObservedRunningTime="2026-07-28 10:35:41.648050738 +0000 UTC m=+732.690142891"
Jul 28 10:37:04.312959 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.312873    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"]
Jul 28 10:37:04.315970 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.315954    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:04.320598 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.320572    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 10:37:04.320729 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.320575    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 10:37:04.320729 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.320638    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gvcl2\""
Jul 28 10:37:04.321040 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.321022    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"]
Jul 28 10:37:04.455524 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.455485    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872m5\" (UniqueName: \"kubernetes.io/projected/7921be03-7225-423e-9007-fd31372340e4-kube-api-access-872m5\") pod \"authorino-operator-75cc8d5cb7-sb6jc\" (UID: \"7921be03-7225-423e-9007-fd31372340e4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:04.556937 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.556891    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872m5\" (UniqueName: \"kubernetes.io/projected/7921be03-7225-423e-9007-fd31372340e4-kube-api-access-872m5\") pod \"authorino-operator-75cc8d5cb7-sb6jc\" (UID: \"7921be03-7225-423e-9007-fd31372340e4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:04.566437 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.566368    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-872m5\" (UniqueName: \"kubernetes.io/projected/7921be03-7225-423e-9007-fd31372340e4-kube-api-access-872m5\") pod \"authorino-operator-75cc8d5cb7-sb6jc\" (UID: \"7921be03-7225-423e-9007-fd31372340e4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:04.626060 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.626014    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:04.760668 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.760537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"]
Jul 28 10:37:04.939618 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:04.939578    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc" event={"ID":"7921be03-7225-423e-9007-fd31372340e4","Type":"ContainerStarted","Data":"a844e119d491cb12ae06abc2280e83bc604fc70f01757d3225cb4f73f87c8a45"}
Jul 28 10:37:07.955065 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:07.955030    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc" event={"ID":"7921be03-7225-423e-9007-fd31372340e4","Type":"ContainerStarted","Data":"6dce7923f1595687df9a21db4615be353801b79ba33c947f1f9ab365b013f514"}
Jul 28 10:37:07.955494 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:07.955132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:07.971510 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:07.971452    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc" podStartSLOduration=1.256104038 podStartE2EDuration="3.971436105s" totalImagesPullingTime="2.715332067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:07.970697657 +0000 UTC m=+819.012789811" watchObservedRunningTime="2026-07-28 10:37:07.971436105 +0000 UTC m=+819.013528258"
Jul 28 10:37:08.947452 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:08.947417    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"]
Jul 28 10:37:08.950620 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:08.950602    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:08.956469 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:08.956442    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-65pzk\""
Jul 28 10:37:08.956861 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:08.956473    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 10:37:08.965228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:08.965200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"]
Jul 28 10:37:09.096947 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:09.096909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt7qv\" (UniqueName: \"kubernetes.io/projected/3fc190f1-a000-4bb3-9d41-0bbabf2d4854-kube-api-access-lt7qv\") pod \"dns-operator-controller-manager-864dd78c4c-xkfdg\" (UID: \"3fc190f1-a000-4bb3-9d41-0bbabf2d4854\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:09.197837 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:09.197751    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt7qv\" (UniqueName: \"kubernetes.io/projected/3fc190f1-a000-4bb3-9d41-0bbabf2d4854-kube-api-access-lt7qv\") pod \"dns-operator-controller-manager-864dd78c4c-xkfdg\" (UID: \"3fc190f1-a000-4bb3-9d41-0bbabf2d4854\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:09.206871 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:09.206842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt7qv\" (UniqueName: \"kubernetes.io/projected/3fc190f1-a000-4bb3-9d41-0bbabf2d4854-kube-api-access-lt7qv\") pod \"dns-operator-controller-manager-864dd78c4c-xkfdg\" (UID: \"3fc190f1-a000-4bb3-9d41-0bbabf2d4854\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:09.261479 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:09.261439    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:09.395763 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:09.395737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"]
Jul 28 10:37:09.398421 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:37:09.398389    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc190f1_a000_4bb3_9d41_0bbabf2d4854.slice/crio-2e4e89838967f59532b18596fd70e4844430980878a00b13e5f2dd44bc172416 WatchSource:0}: Error finding container 2e4e89838967f59532b18596fd70e4844430980878a00b13e5f2dd44bc172416: Status 404 returned error can't find the container with id 2e4e89838967f59532b18596fd70e4844430980878a00b13e5f2dd44bc172416
Jul 28 10:37:09.965171 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:09.965136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg" event={"ID":"3fc190f1-a000-4bb3-9d41-0bbabf2d4854","Type":"ContainerStarted","Data":"2e4e89838967f59532b18596fd70e4844430980878a00b13e5f2dd44bc172416"}
Jul 28 10:37:10.660109 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.660068    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"]
Jul 28 10:37:10.663563 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.663528    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.666046 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.666001    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 28 10:37:10.666250 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.666227    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 28 10:37:10.666392 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.666295    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-f66n2\""
Jul 28 10:37:10.673898 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.672886    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"]
Jul 28 10:37:10.812531 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.812494    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkcqm\" (UniqueName: \"kubernetes.io/projected/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-kube-api-access-pkcqm\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.812683 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.812542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.812683 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.812567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.913964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.913863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.913964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.913910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.914140 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.913990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkcqm\" (UniqueName: \"kubernetes.io/projected/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-kube-api-access-pkcqm\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.914140 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:37:10.914093    2578 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 28 10:37:10.914203 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:37:10.914159    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-plugin-serving-cert podName:9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb nodeName:}" failed. No retries permitted until 2026-07-28 10:37:11.414144043 +0000 UTC m=+822.456236174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-5jsnm" (UID: "9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb") : secret "plugin-serving-cert" not found
Jul 28 10:37:10.914685 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.914667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:10.928396 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:10.928364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkcqm\" (UniqueName: \"kubernetes.io/projected/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-kube-api-access-pkcqm\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:11.419381 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.419346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:11.422039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.422016    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-5jsnm\" (UID: \"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:11.576289 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.576241    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"
Jul 28 10:37:11.713395 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.713365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm"]
Jul 28 10:37:11.715602 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:37:11.715580    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ec1ce85_7bb7_4ae5_abd4_9f1f1a5b60bb.slice/crio-97756a4c23a98fb0efc6c2e7d57fb8aa29f00294e50635eef7dd45eddfe14817 WatchSource:0}: Error finding container 97756a4c23a98fb0efc6c2e7d57fb8aa29f00294e50635eef7dd45eddfe14817: Status 404 returned error can't find the container with id 97756a4c23a98fb0efc6c2e7d57fb8aa29f00294e50635eef7dd45eddfe14817
Jul 28 10:37:11.975248 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.975204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg" event={"ID":"3fc190f1-a000-4bb3-9d41-0bbabf2d4854","Type":"ContainerStarted","Data":"a7db3005d87a0cce445ec459030f4c7ad99c8fdd27cb2a18439112b5351411b2"}
Jul 28 10:37:11.975489 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.975397    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:11.976445 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.976421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm" event={"ID":"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb","Type":"ContainerStarted","Data":"97756a4c23a98fb0efc6c2e7d57fb8aa29f00294e50635eef7dd45eddfe14817"}
Jul 28 10:37:11.994723 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:11.994670    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg" podStartSLOduration=1.989625252 podStartE2EDuration="3.994653904s" totalImagesPullingTime="2.005028652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:11.993620089 +0000 UTC m=+823.035712243" watchObservedRunningTime="2026-07-28 10:37:11.994653904 +0000 UTC m=+823.036746137"
Jul 28 10:37:18.963069 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:18.963039    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-sb6jc"
Jul 28 10:37:22.984639 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:22.984609    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-xkfdg"
Jul 28 10:37:34.081697 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:34.081655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm" event={"ID":"9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb","Type":"ContainerStarted","Data":"d86ae0716f8325544d1d2c681bc5195a879dc7ed099583bcfd497c9c5a589a2a"}
Jul 28 10:37:34.097971 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:34.097910    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-5jsnm" podStartSLOduration=2.395500603 podStartE2EDuration="24.09789316s" totalImagesPullingTime="21.702392557s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:34.096779868 +0000 UTC m=+845.138872022" watchObservedRunningTime="2026-07-28 10:37:34.09789316 +0000 UTC m=+845.139985313"
Jul 28 10:37:52.036899 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.036867    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-n55vx"]
Jul 28 10:37:52.071553 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.071513    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n55vx"]
Jul 28 10:37:52.071721 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.071621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:37:52.074361 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.074342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hlwj8\""
Jul 28 10:37:52.149824 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.149787    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl5cf\" (UniqueName: \"kubernetes.io/projected/76d7da9f-5025-44b6-922d-7f36615abef0-kube-api-access-bl5cf\") pod \"authorino-58cd955675-n55vx\" (UID: \"76d7da9f-5025-44b6-922d-7f36615abef0\") " pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:37:52.154603 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.154532    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqvtp"]
Jul 28 10:37:52.156752 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.156720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:37:52.164238 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.164215    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqvtp"]
Jul 28 10:37:52.250358 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.250316    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fxbz\" (UniqueName: \"kubernetes.io/projected/07fcf2b7-8062-4a91-bcc1-f9566e69783a-kube-api-access-8fxbz\") pod \"authorino-75d556bd77-pqvtp\" (UID: \"07fcf2b7-8062-4a91-bcc1-f9566e69783a\") " pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:37:52.250527 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.250430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl5cf\" (UniqueName: \"kubernetes.io/projected/76d7da9f-5025-44b6-922d-7f36615abef0-kube-api-access-bl5cf\") pod \"authorino-58cd955675-n55vx\" (UID: \"76d7da9f-5025-44b6-922d-7f36615abef0\") " pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:37:52.259069 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.259042    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl5cf\" (UniqueName: \"kubernetes.io/projected/76d7da9f-5025-44b6-922d-7f36615abef0-kube-api-access-bl5cf\") pod \"authorino-58cd955675-n55vx\" (UID: \"76d7da9f-5025-44b6-922d-7f36615abef0\") " pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:37:52.351198 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.351101    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fxbz\" (UniqueName: \"kubernetes.io/projected/07fcf2b7-8062-4a91-bcc1-f9566e69783a-kube-api-access-8fxbz\") pod \"authorino-75d556bd77-pqvtp\" (UID: \"07fcf2b7-8062-4a91-bcc1-f9566e69783a\") " pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:37:52.360182 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.360150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fxbz\" (UniqueName: \"kubernetes.io/projected/07fcf2b7-8062-4a91-bcc1-f9566e69783a-kube-api-access-8fxbz\") pod \"authorino-75d556bd77-pqvtp\" (UID: \"07fcf2b7-8062-4a91-bcc1-f9566e69783a\") " pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:37:52.380509 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.380469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:37:52.465944 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.465849    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:37:52.520083 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.520030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-n55vx"]
Jul 28 10:37:52.523724 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:37:52.523695    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76d7da9f_5025_44b6_922d_7f36615abef0.slice/crio-42d72a88cff25405d0ff47471106caeaaaa3f4aa2421bcbe3773c68ff26ac9d8 WatchSource:0}: Error finding container 42d72a88cff25405d0ff47471106caeaaaa3f4aa2421bcbe3773c68ff26ac9d8: Status 404 returned error can't find the container with id 42d72a88cff25405d0ff47471106caeaaaa3f4aa2421bcbe3773c68ff26ac9d8
Jul 28 10:37:52.610030 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:52.609998    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqvtp"]
Jul 28 10:37:52.612482 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:37:52.612453    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07fcf2b7_8062_4a91_bcc1_f9566e69783a.slice/crio-5f300086c9b57d9083ddd1f315e72d62d09a2d38d4e091e98a15d8358000b020 WatchSource:0}: Error finding container 5f300086c9b57d9083ddd1f315e72d62d09a2d38d4e091e98a15d8358000b020: Status 404 returned error can't find the container with id 5f300086c9b57d9083ddd1f315e72d62d09a2d38d4e091e98a15d8358000b020
Jul 28 10:37:53.168723 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:53.168638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n55vx" event={"ID":"76d7da9f-5025-44b6-922d-7f36615abef0","Type":"ContainerStarted","Data":"42d72a88cff25405d0ff47471106caeaaaa3f4aa2421bcbe3773c68ff26ac9d8"}
Jul 28 10:37:53.170535 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:53.170479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqvtp" event={"ID":"07fcf2b7-8062-4a91-bcc1-f9566e69783a","Type":"ContainerStarted","Data":"5f300086c9b57d9083ddd1f315e72d62d09a2d38d4e091e98a15d8358000b020"}
Jul 28 10:37:56.191778 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:56.191736    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqvtp" event={"ID":"07fcf2b7-8062-4a91-bcc1-f9566e69783a","Type":"ContainerStarted","Data":"f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197"}
Jul 28 10:37:56.207413 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:56.207362    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-pqvtp" podStartSLOduration=0.776335489 podStartE2EDuration="4.207345401s" totalImagesPullingTime="3.431009912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:37:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:37:56.205502975 +0000 UTC m=+867.247595128" watchObservedRunningTime="2026-07-28 10:37:56.207345401 +0000 UTC m=+867.249437554"
Jul 28 10:37:56.232895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:37:56.232856    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n55vx"]
Jul 28 10:38:03.187351 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:38:03.187258    2578 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:0607df08ba8e0fd3f033f93c7b043a2f0693221cd4e9c4a834431a51ae5c87e3: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="quay.io/kuadrant/authorino:v0.24.0"
Jul 28 10:38:03.187714 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:38:03.187427    2578 kuberuntime_manager.go:1459] "Unhandled Error" err="container authorino start failed in pod authorino-58cd955675-n55vx_kuadrant-system(76d7da9f-5025-44b6-922d-7f36615abef0): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:0607df08ba8e0fd3f033f93c7b043a2f0693221cd4e9c4a834431a51ae5c87e3: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 28 10:38:03.187714 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:38:03.187461    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"authorino\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: reading blob sha256:0607df08ba8e0fd3f033f93c7b043a2f0693221cd4e9c4a834431a51ae5c87e3: fetching blob: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="kuadrant-system/authorino-58cd955675-n55vx" podUID="76d7da9f-5025-44b6-922d-7f36615abef0"
Jul 28 10:38:03.430444 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:03.430401    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:38:03.559519 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:03.559441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bl5cf\" (UniqueName: \"kubernetes.io/projected/76d7da9f-5025-44b6-922d-7f36615abef0-kube-api-access-bl5cf\") pod \"76d7da9f-5025-44b6-922d-7f36615abef0\" (UID: \"76d7da9f-5025-44b6-922d-7f36615abef0\") "
Jul 28 10:38:03.561855 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:03.561822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d7da9f-5025-44b6-922d-7f36615abef0-kube-api-access-bl5cf" (OuterVolumeSpecName: "kube-api-access-bl5cf") pod "76d7da9f-5025-44b6-922d-7f36615abef0" (UID: "76d7da9f-5025-44b6-922d-7f36615abef0"). InnerVolumeSpecName "kube-api-access-bl5cf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:03.660650 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:03.660610    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl5cf\" (UniqueName: \"kubernetes.io/projected/76d7da9f-5025-44b6-922d-7f36615abef0-kube-api-access-bl5cf\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:38:04.226159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:04.226126    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-n55vx"
Jul 28 10:38:04.226159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:04.226152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-n55vx" event={"ID":"76d7da9f-5025-44b6-922d-7f36615abef0","Type":"ContainerDied","Data":"42d72a88cff25405d0ff47471106caeaaaa3f4aa2421bcbe3773c68ff26ac9d8"}
Jul 28 10:38:04.257166 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:04.257131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-n55vx"]
Jul 28 10:38:04.260591 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:04.260559    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-n55vx"]
Jul 28 10:38:05.493908 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:05.493876    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76d7da9f-5025-44b6-922d-7f36615abef0" path="/var/lib/kubelet/pods/76d7da9f-5025-44b6-922d-7f36615abef0/volumes"
Jul 28 10:38:19.932749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:19.932668    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w9kdm"]
Jul 28 10:38:20.052762 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.052718    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w9kdm"]
Jul 28 10:38:20.052926 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.052839    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:20.103601 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.103569    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8j5s\" (UniqueName: \"kubernetes.io/projected/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e-kube-api-access-s8j5s\") pod \"authorino-5d4b67c446-w9kdm\" (UID: \"3dc595b8-f718-4c4d-afb2-c0cc74efeb9e\") " pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:20.179573 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.179533    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w9kdm"]
Jul 28 10:38:20.179808 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:38:20.179788    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-s8j5s], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-w9kdm" podUID="3dc595b8-f718-4c4d-afb2-c0cc74efeb9e"
Jul 28 10:38:20.204008 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.203919    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8j5s\" (UniqueName: \"kubernetes.io/projected/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e-kube-api-access-s8j5s\") pod \"authorino-5d4b67c446-w9kdm\" (UID: \"3dc595b8-f718-4c4d-afb2-c0cc74efeb9e\") " pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:20.209568 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.209538    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-65595c8696-95mvx"]
Jul 28 10:38:20.211715 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.211697    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:20.213812 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.213790    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8j5s\" (UniqueName: \"kubernetes.io/projected/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e-kube-api-access-s8j5s\") pod \"authorino-5d4b67c446-w9kdm\" (UID: \"3dc595b8-f718-4c4d-afb2-c0cc74efeb9e\") " pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:20.222293 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.222251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-65595c8696-95mvx"]
Jul 28 10:38:20.293608 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.293575    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:20.299211 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.299186    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:20.304522 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.304499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8j5s\" (UniqueName: \"kubernetes.io/projected/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e-kube-api-access-s8j5s\") pod \"3dc595b8-f718-4c4d-afb2-c0cc74efeb9e\" (UID: \"3dc595b8-f718-4c4d-afb2-c0cc74efeb9e\") "
Jul 28 10:38:20.304690 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.304673    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7xk\" (UniqueName: \"kubernetes.io/projected/fe879528-3a56-44eb-acd2-1dc4f8a3dba0-kube-api-access-gb7xk\") pod \"authorino-65595c8696-95mvx\" (UID: \"fe879528-3a56-44eb-acd2-1dc4f8a3dba0\") " pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:20.306682 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.306658    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e-kube-api-access-s8j5s" (OuterVolumeSpecName: "kube-api-access-s8j5s") pod "3dc595b8-f718-4c4d-afb2-c0cc74efeb9e" (UID: "3dc595b8-f718-4c4d-afb2-c0cc74efeb9e"). InnerVolumeSpecName "kube-api-access-s8j5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:20.405674 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.405634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7xk\" (UniqueName: \"kubernetes.io/projected/fe879528-3a56-44eb-acd2-1dc4f8a3dba0-kube-api-access-gb7xk\") pod \"authorino-65595c8696-95mvx\" (UID: \"fe879528-3a56-44eb-acd2-1dc4f8a3dba0\") " pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:20.405852 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.405711    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8j5s\" (UniqueName: \"kubernetes.io/projected/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e-kube-api-access-s8j5s\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:38:20.414343 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.414317    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7xk\" (UniqueName: \"kubernetes.io/projected/fe879528-3a56-44eb-acd2-1dc4f8a3dba0-kube-api-access-gb7xk\") pod \"authorino-65595c8696-95mvx\" (UID: \"fe879528-3a56-44eb-acd2-1dc4f8a3dba0\") " pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:20.474148 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.474072    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65595c8696-95mvx"]
Jul 28 10:38:20.474348 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.474334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:20.501520 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.501489    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d8779c896-lpcw2"]
Jul 28 10:38:20.504170 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.504146    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.507086 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.507064    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 10:38:20.515513 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.515489    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d8779c896-lpcw2"]
Jul 28 10:38:20.603003 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.602971    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65595c8696-95mvx"]
Jul 28 10:38:20.605967 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:38:20.605945    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe879528_3a56_44eb_acd2_1dc4f8a3dba0.slice/crio-4a5fac1b5f7b1a08c63f0956e5ce84c0ec96f3a48c26368fcc50069c59906c88 WatchSource:0}: Error finding container 4a5fac1b5f7b1a08c63f0956e5ce84c0ec96f3a48c26368fcc50069c59906c88: Status 404 returned error can't find the container with id 4a5fac1b5f7b1a08c63f0956e5ce84c0ec96f3a48c26368fcc50069c59906c88
Jul 28 10:38:20.607009 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.606984    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glv4c\" (UniqueName: \"kubernetes.io/projected/93540afa-f825-4b0c-a475-3fd44f9a95cb-kube-api-access-glv4c\") pod \"authorino-5d8779c896-lpcw2\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") " pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.607131 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.607067    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93540afa-f825-4b0c-a475-3fd44f9a95cb-tls-cert\") pod \"authorino-5d8779c896-lpcw2\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") " pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.707972 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.707935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glv4c\" (UniqueName: \"kubernetes.io/projected/93540afa-f825-4b0c-a475-3fd44f9a95cb-kube-api-access-glv4c\") pod \"authorino-5d8779c896-lpcw2\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") " pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.708164 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.707989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93540afa-f825-4b0c-a475-3fd44f9a95cb-tls-cert\") pod \"authorino-5d8779c896-lpcw2\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") " pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.710558 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.710535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93540afa-f825-4b0c-a475-3fd44f9a95cb-tls-cert\") pod \"authorino-5d8779c896-lpcw2\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") " pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.715987 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.715964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glv4c\" (UniqueName: \"kubernetes.io/projected/93540afa-f825-4b0c-a475-3fd44f9a95cb-kube-api-access-glv4c\") pod \"authorino-5d8779c896-lpcw2\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") " pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.815212 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.815127    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:38:20.948017 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:20.947884    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d8779c896-lpcw2"]
Jul 28 10:38:20.950153 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:38:20.950126    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93540afa_f825_4b0c_a475_3fd44f9a95cb.slice/crio-a641fc3b8a9ec282090a089961582c39c1bc70405922533a31a41121adf74929 WatchSource:0}: Error finding container a641fc3b8a9ec282090a089961582c39c1bc70405922533a31a41121adf74929: Status 404 returned error can't find the container with id a641fc3b8a9ec282090a089961582c39c1bc70405922533a31a41121adf74929
Jul 28 10:38:21.303935 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.303896    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d8779c896-lpcw2" event={"ID":"93540afa-f825-4b0c-a475-3fd44f9a95cb","Type":"ContainerStarted","Data":"a641fc3b8a9ec282090a089961582c39c1bc70405922533a31a41121adf74929"}
Jul 28 10:38:21.305526 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.305496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65595c8696-95mvx" event={"ID":"fe879528-3a56-44eb-acd2-1dc4f8a3dba0","Type":"ContainerStarted","Data":"49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb"}
Jul 28 10:38:21.305526 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.305523    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-w9kdm"
Jul 28 10:38:21.305688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.305550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65595c8696-95mvx" event={"ID":"fe879528-3a56-44eb-acd2-1dc4f8a3dba0","Type":"ContainerStarted","Data":"4a5fac1b5f7b1a08c63f0956e5ce84c0ec96f3a48c26368fcc50069c59906c88"}
Jul 28 10:38:21.305688 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.305625    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-65595c8696-95mvx" podUID="fe879528-3a56-44eb-acd2-1dc4f8a3dba0" containerName="authorino" containerID="cri-o://49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb" gracePeriod=30
Jul 28 10:38:21.321684 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.321641    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-65595c8696-95mvx" podStartSLOduration=0.961174748 podStartE2EDuration="1.32162692s" totalImagesPullingTime="360.452172ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:21.319938407 +0000 UTC m=+892.362030559" watchObservedRunningTime="2026-07-28 10:38:21.32162692 +0000 UTC m=+892.363719072"
Jul 28 10:38:21.375470 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.375431    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w9kdm"]
Jul 28 10:38:21.379875 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.379845    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-w9kdm"]
Jul 28 10:38:21.496088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.496016    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3dc595b8-f718-4c4d-afb2-c0cc74efeb9e" path="/var/lib/kubelet/pods/3dc595b8-f718-4c4d-afb2-c0cc74efeb9e/volumes"
Jul 28 10:38:21.554355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.554329    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:21.617073 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.617036    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gb7xk\" (UniqueName: \"kubernetes.io/projected/fe879528-3a56-44eb-acd2-1dc4f8a3dba0-kube-api-access-gb7xk\") pod \"fe879528-3a56-44eb-acd2-1dc4f8a3dba0\" (UID: \"fe879528-3a56-44eb-acd2-1dc4f8a3dba0\") "
Jul 28 10:38:21.619413 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.619388    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe879528-3a56-44eb-acd2-1dc4f8a3dba0-kube-api-access-gb7xk" (OuterVolumeSpecName: "kube-api-access-gb7xk") pod "fe879528-3a56-44eb-acd2-1dc4f8a3dba0" (UID: "fe879528-3a56-44eb-acd2-1dc4f8a3dba0"). InnerVolumeSpecName "kube-api-access-gb7xk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:21.718580 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:21.718532    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7xk\" (UniqueName: \"kubernetes.io/projected/fe879528-3a56-44eb-acd2-1dc4f8a3dba0-kube-api-access-gb7xk\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:38:22.311547 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.311443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d8779c896-lpcw2" event={"ID":"93540afa-f825-4b0c-a475-3fd44f9a95cb","Type":"ContainerStarted","Data":"1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988"}
Jul 28 10:38:22.312611 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.312585    2578 generic.go:356] "Generic (PLEG): container finished" podID="fe879528-3a56-44eb-acd2-1dc4f8a3dba0" containerID="49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb" exitCode=0
Jul 28 10:38:22.312738 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.312625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65595c8696-95mvx" event={"ID":"fe879528-3a56-44eb-acd2-1dc4f8a3dba0","Type":"ContainerDied","Data":"49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb"}
Jul 28 10:38:22.312738 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.312669    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-65595c8696-95mvx"
Jul 28 10:38:22.312738 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.312706    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-65595c8696-95mvx" event={"ID":"fe879528-3a56-44eb-acd2-1dc4f8a3dba0","Type":"ContainerDied","Data":"4a5fac1b5f7b1a08c63f0956e5ce84c0ec96f3a48c26368fcc50069c59906c88"}
Jul 28 10:38:22.312738 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.312735    2578 scope.go:117] "RemoveContainer" containerID="49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb"
Jul 28 10:38:22.321325 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.321304    2578 scope.go:117] "RemoveContainer" containerID="49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb"
Jul 28 10:38:22.321614 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:38:22.321594    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb\": container with ID starting with 49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb not found: ID does not exist" containerID="49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb"
Jul 28 10:38:22.321668 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.321624    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb"} err="failed to get container status \"49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb\": rpc error: code = NotFound desc = could not find container \"49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb\": container with ID starting with 49c6786cf2d0c6ad690aa75f3244cf348067933c2dda8e69df89c1cfc1821ffb not found: ID does not exist"
Jul 28 10:38:22.326321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.326257    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d8779c896-lpcw2" podStartSLOduration=2.02355543 podStartE2EDuration="2.326239988s" totalImagesPullingTime="302.684558ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:22.325710977 +0000 UTC m=+893.367803124" watchObservedRunningTime="2026-07-28 10:38:22.326239988 +0000 UTC m=+893.368332141"
Jul 28 10:38:22.341307 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.341258    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-65595c8696-95mvx"]
Jul 28 10:38:22.345948 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.345921    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-65595c8696-95mvx"]
Jul 28 10:38:22.352046 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.352020    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqvtp"]
Jul 28 10:38:22.352253 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.352227    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-pqvtp" podUID="07fcf2b7-8062-4a91-bcc1-f9566e69783a" containerName="authorino" containerID="cri-o://f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197" gracePeriod=30
Jul 28 10:38:22.602381 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.602358    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:38:22.627066 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.627034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fxbz\" (UniqueName: \"kubernetes.io/projected/07fcf2b7-8062-4a91-bcc1-f9566e69783a-kube-api-access-8fxbz\") pod \"07fcf2b7-8062-4a91-bcc1-f9566e69783a\" (UID: \"07fcf2b7-8062-4a91-bcc1-f9566e69783a\") "
Jul 28 10:38:22.629405 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.629374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fcf2b7-8062-4a91-bcc1-f9566e69783a-kube-api-access-8fxbz" (OuterVolumeSpecName: "kube-api-access-8fxbz") pod "07fcf2b7-8062-4a91-bcc1-f9566e69783a" (UID: "07fcf2b7-8062-4a91-bcc1-f9566e69783a"). InnerVolumeSpecName "kube-api-access-8fxbz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:38:22.728258 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:22.728219    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fxbz\" (UniqueName: \"kubernetes.io/projected/07fcf2b7-8062-4a91-bcc1-f9566e69783a-kube-api-access-8fxbz\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:38:23.318291 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.318238    2578 generic.go:356] "Generic (PLEG): container finished" podID="07fcf2b7-8062-4a91-bcc1-f9566e69783a" containerID="f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197" exitCode=0
Jul 28 10:38:23.318775 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.318324    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-pqvtp"
Jul 28 10:38:23.318775 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.318326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqvtp" event={"ID":"07fcf2b7-8062-4a91-bcc1-f9566e69783a","Type":"ContainerDied","Data":"f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197"}
Jul 28 10:38:23.318775 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.318374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-pqvtp" event={"ID":"07fcf2b7-8062-4a91-bcc1-f9566e69783a","Type":"ContainerDied","Data":"5f300086c9b57d9083ddd1f315e72d62d09a2d38d4e091e98a15d8358000b020"}
Jul 28 10:38:23.318775 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.318397    2578 scope.go:117] "RemoveContainer" containerID="f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197"
Jul 28 10:38:23.328431 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.328412    2578 scope.go:117] "RemoveContainer" containerID="f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197"
Jul 28 10:38:23.328802 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:38:23.328772    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197\": container with ID starting with f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197 not found: ID does not exist" containerID="f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197"
Jul 28 10:38:23.328899 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.328812    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197"} err="failed to get container status \"f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197\": rpc error: code = NotFound desc = could not find container \"f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197\": container with ID starting with f90ee325f82d6989306b84dec0023d303f3919c796c9dc04982a45e1789df197 not found: ID does not exist"
Jul 28 10:38:23.339332 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.339308    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqvtp"]
Jul 28 10:38:23.343149 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.343127    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-pqvtp"]
Jul 28 10:38:23.493675 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.493641    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07fcf2b7-8062-4a91-bcc1-f9566e69783a" path="/var/lib/kubelet/pods/07fcf2b7-8062-4a91-bcc1-f9566e69783a/volumes"
Jul 28 10:38:23.493955 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:23.493943    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe879528-3a56-44eb-acd2-1dc4f8a3dba0" path="/var/lib/kubelet/pods/fe879528-3a56-44eb-acd2-1dc4f8a3dba0/volumes"
Jul 28 10:38:27.245748 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.245713    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-qspnk"]
Jul 28 10:38:27.246159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.246005    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fcf2b7-8062-4a91-bcc1-f9566e69783a" containerName="authorino"
Jul 28 10:38:27.246159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.246015    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fcf2b7-8062-4a91-bcc1-f9566e69783a" containerName="authorino"
Jul 28 10:38:27.246159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.246028    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe879528-3a56-44eb-acd2-1dc4f8a3dba0" containerName="authorino"
Jul 28 10:38:27.246159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.246033    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe879528-3a56-44eb-acd2-1dc4f8a3dba0" containerName="authorino"
Jul 28 10:38:27.246159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.246086    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fcf2b7-8062-4a91-bcc1-f9566e69783a" containerName="authorino"
Jul 28 10:38:27.246159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.246095    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe879528-3a56-44eb-acd2-1dc4f8a3dba0" containerName="authorino"
Jul 28 10:38:27.250104 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.250084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.252798 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.252778    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 10:38:27.254045 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.254007    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 10:38:27.254045 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.254038    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rktk6\""
Jul 28 10:38:27.258955 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.258936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-qspnk"]
Jul 28 10:38:27.368323 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.368291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d941fa37-034f-43a8-ab12-e8a2f640ff7a-webhook-cert\") pod \"maas-controller-84d7d4f6df-qspnk\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") " pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.368500 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.368332    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb8gx\" (UniqueName: \"kubernetes.io/projected/d941fa37-034f-43a8-ab12-e8a2f640ff7a-kube-api-access-rb8gx\") pod \"maas-controller-84d7d4f6df-qspnk\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") " pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.469136 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.469106    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d941fa37-034f-43a8-ab12-e8a2f640ff7a-webhook-cert\") pod \"maas-controller-84d7d4f6df-qspnk\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") " pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.469319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.469142    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb8gx\" (UniqueName: \"kubernetes.io/projected/d941fa37-034f-43a8-ab12-e8a2f640ff7a-kube-api-access-rb8gx\") pod \"maas-controller-84d7d4f6df-qspnk\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") " pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.471793 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.471771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d941fa37-034f-43a8-ab12-e8a2f640ff7a-webhook-cert\") pod \"maas-controller-84d7d4f6df-qspnk\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") " pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.477796 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.477774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb8gx\" (UniqueName: \"kubernetes.io/projected/d941fa37-034f-43a8-ab12-e8a2f640ff7a-kube-api-access-rb8gx\") pod \"maas-controller-84d7d4f6df-qspnk\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") " pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.561194 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.561111    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:27.694601 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:27.694564    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-qspnk"]
Jul 28 10:38:27.697072 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:38:27.697042    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd941fa37_034f_43a8_ab12_e8a2f640ff7a.slice/crio-8e381d294e82d7b3bf75fd9bf5f524bf5bfbb94a893de2041986f5adef362208 WatchSource:0}: Error finding container 8e381d294e82d7b3bf75fd9bf5f524bf5bfbb94a893de2041986f5adef362208: Status 404 returned error can't find the container with id 8e381d294e82d7b3bf75fd9bf5f524bf5bfbb94a893de2041986f5adef362208
Jul 28 10:38:28.343588 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:28.343550    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-qspnk" event={"ID":"d941fa37-034f-43a8-ab12-e8a2f640ff7a","Type":"ContainerStarted","Data":"8e381d294e82d7b3bf75fd9bf5f524bf5bfbb94a893de2041986f5adef362208"}
Jul 28 10:38:29.833887 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:29.833855    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:38:29.836530 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:29.836510    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:38:29.861083 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:29.861063    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 10:38:30.354834 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:30.354796    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-qspnk" event={"ID":"d941fa37-034f-43a8-ab12-e8a2f640ff7a","Type":"ContainerStarted","Data":"5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712"}
Jul 28 10:38:30.355010 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:30.354889    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:38:30.372476 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:30.372416    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d7d4f6df-qspnk" podStartSLOduration=1.212271281 podStartE2EDuration="3.372397627s" totalImagesPullingTime="2.160126346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:38:30.371499706 +0000 UTC m=+901.413591862" watchObservedRunningTime="2026-07-28 10:38:30.372397627 +0000 UTC m=+901.414489781"
Jul 28 10:38:41.361595 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:38:41.361563    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:39:24.643209 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.643171    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"]
Jul 28 10:39:24.646455 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.646432    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.649934 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.649903    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 10:39:24.650077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.649949    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 10:39:24.650077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.649923    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-65s9d\""
Jul 28 10:39:24.650077 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.649959    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 28 10:39:24.656632 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.656603    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"]
Jul 28 10:39:24.749892 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.749859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.750070 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.749914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.750070 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.749994    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.750070 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.750061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.750196 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.750089    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp2jc\" (UniqueName: \"kubernetes.io/projected/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-kube-api-access-xp2jc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.750196 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.750120    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.850621 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.850579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.850806 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.850638    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.850806 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.850711    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.850806 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.850744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp2jc\" (UniqueName: \"kubernetes.io/projected/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-kube-api-access-xp2jc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.850806 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.850775    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.851008 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.850806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.851089 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.851059    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.851089 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.851077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.851340 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.851316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.853312 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.853255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.853579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.853560    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.859011 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.858978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp2jc\" (UniqueName: \"kubernetes.io/projected/d3b6a563-3d5c-4ec4-bd14-fa32a90117e7-kube-api-access-xp2jc\") pod \"e2e-distinct-simulated-kserve-59975d69f9-vtkvj\" (UID: \"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:24.956954 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:24.956903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:25.100021 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:25.099993    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"]
Jul 28 10:39:25.102139 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:39:25.102109    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b6a563_3d5c_4ec4_bd14_fa32a90117e7.slice/crio-9003b0e80f6ab588582e325c3cd29dd6e84a28ce59afd7209dfdd0a68824e823 WatchSource:0}: Error finding container 9003b0e80f6ab588582e325c3cd29dd6e84a28ce59afd7209dfdd0a68824e823: Status 404 returned error can't find the container with id 9003b0e80f6ab588582e325c3cd29dd6e84a28ce59afd7209dfdd0a68824e823
Jul 28 10:39:25.590624 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:25.590574    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj" event={"ID":"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7","Type":"ContainerStarted","Data":"9003b0e80f6ab588582e325c3cd29dd6e84a28ce59afd7209dfdd0a68824e823"}
Jul 28 10:39:26.596195 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:26.596097    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj" event={"ID":"d3b6a563-3d5c-4ec4-bd14-fa32a90117e7","Type":"ContainerStarted","Data":"904e55e5ff273347cac1ca3dc1d133dd805035c7067ab24fb5dc75dfaa7fa112"}
Jul 28 10:39:26.613300 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:26.613228    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj" podStartSLOduration=1.446909165 podStartE2EDuration="2.613210407s" totalImagesPullingTime="1.166301242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:26.611807488 +0000 UTC m=+957.653899642" watchObservedRunningTime="2026-07-28 10:39:26.613210407 +0000 UTC m=+957.655302559"
Jul 28 10:39:30.242776 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.242728    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"]
Jul 28 10:39:30.246203 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.246178    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.248786 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.248757    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 10:39:30.255114 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.255082    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"]
Jul 28 10:39:30.297669 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.297625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.297669 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.297667    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.297894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.297686    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.297894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.297774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.297894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.297804    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.297894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.297830    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftdw\" (UniqueName: \"kubernetes.io/projected/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-kube-api-access-5ftdw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399318 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399517 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399392    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftdw\" (UniqueName: \"kubernetes.io/projected/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-kube-api-access-5ftdw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399685 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399540    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399685 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399685 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399655    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399828 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399757    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.399892 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.399875    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.401837 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.401815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.402015 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.401996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.408343 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.408310    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftdw\" (UniqueName: \"kubernetes.io/projected/7753c7ba-d971-4c0d-93e5-14c09b44cdaf-kube-api-access-5ftdw\") pod \"e2e-trlp-test-simulated-kserve-598466548d-k86sz\" (UID: \"7753c7ba-d971-4c0d-93e5-14c09b44cdaf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.558017 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.557913    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:30.697262 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:30.697222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"]
Jul 28 10:39:30.701081 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:39:30.701051    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7753c7ba_d971_4c0d_93e5_14c09b44cdaf.slice/crio-137bac44e7430a51bfd7f6ccee750d3bd48f437bdc600012c0d88ae64dba5af1 WatchSource:0}: Error finding container 137bac44e7430a51bfd7f6ccee750d3bd48f437bdc600012c0d88ae64dba5af1: Status 404 returned error can't find the container with id 137bac44e7430a51bfd7f6ccee750d3bd48f437bdc600012c0d88ae64dba5af1
Jul 28 10:39:31.623195 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:31.623157    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz" event={"ID":"7753c7ba-d971-4c0d-93e5-14c09b44cdaf","Type":"ContainerStarted","Data":"fe88626f636628b82cd74635fe6653ca3cabf7bd4eb2052d85cef316160037d1"}
Jul 28 10:39:31.623692 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:31.623215    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz" event={"ID":"7753c7ba-d971-4c0d-93e5-14c09b44cdaf","Type":"ContainerStarted","Data":"137bac44e7430a51bfd7f6ccee750d3bd48f437bdc600012c0d88ae64dba5af1"}
Jul 28 10:39:31.640794 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:31.640721    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz" podStartSLOduration=1.338360983 podStartE2EDuration="1.640698873s" totalImagesPullingTime="302.33789ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:39:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:39:31.640552333 +0000 UTC m=+962.682644487" watchObservedRunningTime="2026-07-28 10:39:31.640698873 +0000 UTC m=+962.682791030"
Jul 28 10:39:34.958039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:34.957995    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:34.958460 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:34.958073    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:34.970908 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:34.970881    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:35.652016 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:35.651987    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-vtkvj"
Jul 28 10:39:40.559014 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:40.558977    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:40.559536 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:40.559066    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:40.571621 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:40.571591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:39:40.674385 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:39:40.674352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-k86sz"
Jul 28 10:40:55.332835 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.332796    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54bc6dc669-9pw7h"]
Jul 28 10:40:55.336017 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.335997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.345717 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.345689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54bc6dc669-9pw7h"]
Jul 28 10:40:55.427463 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.427424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c-tls-cert\") pod \"authorino-54bc6dc669-9pw7h\" (UID: \"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c\") " pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.427661 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.427542    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsnc2\" (UniqueName: \"kubernetes.io/projected/a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c-kube-api-access-gsnc2\") pod \"authorino-54bc6dc669-9pw7h\" (UID: \"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c\") " pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.528742 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.528688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c-tls-cert\") pod \"authorino-54bc6dc669-9pw7h\" (UID: \"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c\") " pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.528932 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.528790    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsnc2\" (UniqueName: \"kubernetes.io/projected/a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c-kube-api-access-gsnc2\") pod \"authorino-54bc6dc669-9pw7h\" (UID: \"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c\") " pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.531368 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.531347    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c-tls-cert\") pod \"authorino-54bc6dc669-9pw7h\" (UID: \"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c\") " pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.537053 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.537027    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsnc2\" (UniqueName: \"kubernetes.io/projected/a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c-kube-api-access-gsnc2\") pod \"authorino-54bc6dc669-9pw7h\" (UID: \"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c\") " pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.644744 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.644652    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54bc6dc669-9pw7h"
Jul 28 10:40:55.776386 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.776357    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54bc6dc669-9pw7h"]
Jul 28 10:40:55.778957 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:40:55.778926    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1d0bc1b_0ec8_47ea_ae79_c5ddcc91805c.slice/crio-aab2f17e3c5cb9105b9ee14897160217c6450ec578618be7e50e35059bef1b88 WatchSource:0}: Error finding container aab2f17e3c5cb9105b9ee14897160217c6450ec578618be7e50e35059bef1b88: Status 404 returned error can't find the container with id aab2f17e3c5cb9105b9ee14897160217c6450ec578618be7e50e35059bef1b88
Jul 28 10:40:55.780255 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.780235    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:40:55.996410 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:55.996374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54bc6dc669-9pw7h" event={"ID":"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c","Type":"ContainerStarted","Data":"aab2f17e3c5cb9105b9ee14897160217c6450ec578618be7e50e35059bef1b88"}
Jul 28 10:40:57.004115 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.004073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-54bc6dc669-9pw7h" event={"ID":"a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c","Type":"ContainerStarted","Data":"1b71cfc6804fedfce68b7334e8b24b8807cc519d0494b965802c2955604be4c4"}
Jul 28 10:40:57.031431 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.031366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-54bc6dc669-9pw7h" podStartSLOduration=1.521426945 podStartE2EDuration="2.031348776s" totalImagesPullingTime="509.921831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:40:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:40:57.029057459 +0000 UTC m=+1048.071149627" watchObservedRunningTime="2026-07-28 10:40:57.031348776 +0000 UTC m=+1048.073440968"
Jul 28 10:40:57.082913 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.082865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d8779c896-lpcw2"]
Jul 28 10:40:57.083673 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.083642    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d8779c896-lpcw2" podUID="93540afa-f825-4b0c-a475-3fd44f9a95cb" containerName="authorino" containerID="cri-o://1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988" gracePeriod=30
Jul 28 10:40:57.352996 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.352965    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:40:57.449648 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.448894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93540afa-f825-4b0c-a475-3fd44f9a95cb-tls-cert\") pod \"93540afa-f825-4b0c-a475-3fd44f9a95cb\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") "
Jul 28 10:40:57.449648 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.448964    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glv4c\" (UniqueName: \"kubernetes.io/projected/93540afa-f825-4b0c-a475-3fd44f9a95cb-kube-api-access-glv4c\") pod \"93540afa-f825-4b0c-a475-3fd44f9a95cb\" (UID: \"93540afa-f825-4b0c-a475-3fd44f9a95cb\") "
Jul 28 10:40:57.454477 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.454416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93540afa-f825-4b0c-a475-3fd44f9a95cb-kube-api-access-glv4c" (OuterVolumeSpecName: "kube-api-access-glv4c") pod "93540afa-f825-4b0c-a475-3fd44f9a95cb" (UID: "93540afa-f825-4b0c-a475-3fd44f9a95cb"). InnerVolumeSpecName "kube-api-access-glv4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:57.473793 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.473734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93540afa-f825-4b0c-a475-3fd44f9a95cb-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "93540afa-f825-4b0c-a475-3fd44f9a95cb" (UID: "93540afa-f825-4b0c-a475-3fd44f9a95cb"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:57.549859 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.549825    2578 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/93540afa-f825-4b0c-a475-3fd44f9a95cb-tls-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:40:57.549859 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:57.549853    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glv4c\" (UniqueName: \"kubernetes.io/projected/93540afa-f825-4b0c-a475-3fd44f9a95cb-kube-api-access-glv4c\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:40:58.012303 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.011700    2578 generic.go:356] "Generic (PLEG): container finished" podID="93540afa-f825-4b0c-a475-3fd44f9a95cb" containerID="1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988" exitCode=0
Jul 28 10:40:58.013331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.012855    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d8779c896-lpcw2"
Jul 28 10:40:58.013331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.012853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d8779c896-lpcw2" event={"ID":"93540afa-f825-4b0c-a475-3fd44f9a95cb","Type":"ContainerDied","Data":"1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988"}
Jul 28 10:40:58.013331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.012983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d8779c896-lpcw2" event={"ID":"93540afa-f825-4b0c-a475-3fd44f9a95cb","Type":"ContainerDied","Data":"a641fc3b8a9ec282090a089961582c39c1bc70405922533a31a41121adf74929"}
Jul 28 10:40:58.013331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.013006    2578 scope.go:117] "RemoveContainer" containerID="1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988"
Jul 28 10:40:58.030565 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.027549    2578 scope.go:117] "RemoveContainer" containerID="1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988"
Jul 28 10:40:58.030565 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:40:58.027918    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988\": container with ID starting with 1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988 not found: ID does not exist" containerID="1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988"
Jul 28 10:40:58.030565 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.027946    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988"} err="failed to get container status \"1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988\": rpc error: code = NotFound desc = could not find container \"1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988\": container with ID starting with 1184a520616cb3470f73564a30ab51e3dd7c2e7080eb4842f621940c49c77988 not found: ID does not exist"
Jul 28 10:40:58.033794 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.033753    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d8779c896-lpcw2"]
Jul 28 10:40:58.039993 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:58.039956    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d8779c896-lpcw2"]
Jul 28 10:40:59.494145 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:40:59.494111    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93540afa-f825-4b0c-a475-3fd44f9a95cb" path="/var/lib/kubelet/pods/93540afa-f825-4b0c-a475-3fd44f9a95cb/volumes"
Jul 28 10:42:00.922882 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:00.922841    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-qspnk"]
Jul 28 10:42:00.923421 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:00.923060    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84d7d4f6df-qspnk" podUID="d941fa37-034f-43a8-ab12-e8a2f640ff7a" containerName="manager" containerID="cri-o://5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712" gracePeriod=30
Jul 28 10:42:01.170172 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.170149    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:42:01.195456 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.195374    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d941fa37-034f-43a8-ab12-e8a2f640ff7a-webhook-cert\") pod \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") "
Jul 28 10:42:01.195625 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.195480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb8gx\" (UniqueName: \"kubernetes.io/projected/d941fa37-034f-43a8-ab12-e8a2f640ff7a-kube-api-access-rb8gx\") pod \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\" (UID: \"d941fa37-034f-43a8-ab12-e8a2f640ff7a\") "
Jul 28 10:42:01.198106 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.198071    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d941fa37-034f-43a8-ab12-e8a2f640ff7a-kube-api-access-rb8gx" (OuterVolumeSpecName: "kube-api-access-rb8gx") pod "d941fa37-034f-43a8-ab12-e8a2f640ff7a" (UID: "d941fa37-034f-43a8-ab12-e8a2f640ff7a"). InnerVolumeSpecName "kube-api-access-rb8gx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:42:01.198601 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.198565    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d941fa37-034f-43a8-ab12-e8a2f640ff7a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d941fa37-034f-43a8-ab12-e8a2f640ff7a" (UID: "d941fa37-034f-43a8-ab12-e8a2f640ff7a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:42:01.296242 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.296202    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb8gx\" (UniqueName: \"kubernetes.io/projected/d941fa37-034f-43a8-ab12-e8a2f640ff7a-kube-api-access-rb8gx\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:42:01.296242 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.296234    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d941fa37-034f-43a8-ab12-e8a2f640ff7a-webhook-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:42:01.302765 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.302738    2578 generic.go:356] "Generic (PLEG): container finished" podID="d941fa37-034f-43a8-ab12-e8a2f640ff7a" containerID="5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712" exitCode=0
Jul 28 10:42:01.302913 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.302783    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-qspnk"
Jul 28 10:42:01.302913 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.302832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-qspnk" event={"ID":"d941fa37-034f-43a8-ab12-e8a2f640ff7a","Type":"ContainerDied","Data":"5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712"}
Jul 28 10:42:01.302913 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.302891    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-qspnk" event={"ID":"d941fa37-034f-43a8-ab12-e8a2f640ff7a","Type":"ContainerDied","Data":"8e381d294e82d7b3bf75fd9bf5f524bf5bfbb94a893de2041986f5adef362208"}
Jul 28 10:42:01.303074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.302923    2578 scope.go:117] "RemoveContainer" containerID="5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712"
Jul 28 10:42:01.311854 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.311836    2578 scope.go:117] "RemoveContainer" containerID="5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712"
Jul 28 10:42:01.312129 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:42:01.312104    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712\": container with ID starting with 5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712 not found: ID does not exist" containerID="5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712"
Jul 28 10:42:01.312192 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.312140    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712"} err="failed to get container status \"5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712\": rpc error: code = NotFound desc = could not find container \"5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712\": container with ID starting with 5abe6a9591df0579e5c53aa38d591880aa71d689f89d150ad1bc75abbb48a712 not found: ID does not exist"
Jul 28 10:42:01.324735 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.324708    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-qspnk"]
Jul 28 10:42:01.328733 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.328711    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-qspnk"]
Jul 28 10:42:01.494228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:42:01.494147    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d941fa37-034f-43a8-ab12-e8a2f640ff7a" path="/var/lib/kubelet/pods/d941fa37-034f-43a8-ab12-e8a2f640ff7a/volumes"
Jul 28 10:43:02.269676 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.269594    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-zdhj5"]
Jul 28 10:43:02.270074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.269910    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d941fa37-034f-43a8-ab12-e8a2f640ff7a" containerName="manager"
Jul 28 10:43:02.270074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.269920    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d941fa37-034f-43a8-ab12-e8a2f640ff7a" containerName="manager"
Jul 28 10:43:02.270074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.269930    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93540afa-f825-4b0c-a475-3fd44f9a95cb" containerName="authorino"
Jul 28 10:43:02.270074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.269936    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="93540afa-f825-4b0c-a475-3fd44f9a95cb" containerName="authorino"
Jul 28 10:43:02.270074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.269998    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d941fa37-034f-43a8-ab12-e8a2f640ff7a" containerName="manager"
Jul 28 10:43:02.270074 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.270006    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="93540afa-f825-4b0c-a475-3fd44f9a95cb" containerName="authorino"
Jul 28 10:43:02.272535 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.272516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.276679 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.276653    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 10:43:02.276817 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.276680    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rktk6\""
Jul 28 10:43:02.276817 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.276655    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 10:43:02.284914 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.284874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-zdhj5"]
Jul 28 10:43:02.299641 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.299596    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-webhook-cert\") pod \"maas-controller-84d7d4f6df-zdhj5\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") " pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.299797 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.299731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsf6r\" (UniqueName: \"kubernetes.io/projected/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-kube-api-access-zsf6r\") pod \"maas-controller-84d7d4f6df-zdhj5\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") " pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.400644 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.400604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsf6r\" (UniqueName: \"kubernetes.io/projected/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-kube-api-access-zsf6r\") pod \"maas-controller-84d7d4f6df-zdhj5\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") " pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.400644 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.400650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-webhook-cert\") pod \"maas-controller-84d7d4f6df-zdhj5\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") " pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.403304 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.403256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-webhook-cert\") pod \"maas-controller-84d7d4f6df-zdhj5\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") " pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.413079 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.413051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsf6r\" (UniqueName: \"kubernetes.io/projected/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-kube-api-access-zsf6r\") pod \"maas-controller-84d7d4f6df-zdhj5\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") " pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.584346 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.584240    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:02.724455 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:02.724420    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-zdhj5"]
Jul 28 10:43:02.727947 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:43:02.727920    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb4efd7c_1575_48aa_b5a8_22ecdbd87729.slice/crio-071607b7254ddc106d0f04fa23af2d2d5aa447629216379d95b332029d624ef7 WatchSource:0}: Error finding container 071607b7254ddc106d0f04fa23af2d2d5aa447629216379d95b332029d624ef7: Status 404 returned error can't find the container with id 071607b7254ddc106d0f04fa23af2d2d5aa447629216379d95b332029d624ef7
Jul 28 10:43:03.570445 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:03.570344    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5" event={"ID":"eb4efd7c-1575-48aa-b5a8-22ecdbd87729","Type":"ContainerStarted","Data":"e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e"}
Jul 28 10:43:03.570445 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:03.570414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5" event={"ID":"eb4efd7c-1575-48aa-b5a8-22ecdbd87729","Type":"ContainerStarted","Data":"071607b7254ddc106d0f04fa23af2d2d5aa447629216379d95b332029d624ef7"}
Jul 28 10:43:03.570445 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:03.570442    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:03.588167 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:03.588108    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5" podStartSLOduration=1.141591036 podStartE2EDuration="1.588090454s" totalImagesPullingTime="446.499418ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:43:03.586128452 +0000 UTC m=+1174.628220606" watchObservedRunningTime="2026-07-28 10:43:03.588090454 +0000 UTC m=+1174.630182606"
Jul 28 10:43:14.577014 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:14.576975    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:19.383696 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.383654    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-zdhj5"]
Jul 28 10:43:19.384164 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.383894    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5" podUID="eb4efd7c-1575-48aa-b5a8-22ecdbd87729" containerName="manager" containerID="cri-o://e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e" gracePeriod=30
Jul 28 10:43:19.634985 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.634916    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:19.643670 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.643637    2578 generic.go:356] "Generic (PLEG): container finished" podID="eb4efd7c-1575-48aa-b5a8-22ecdbd87729" containerID="e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e" exitCode=0
Jul 28 10:43:19.643862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.643684    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5"
Jul 28 10:43:19.643862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.643721    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5" event={"ID":"eb4efd7c-1575-48aa-b5a8-22ecdbd87729","Type":"ContainerDied","Data":"e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e"}
Jul 28 10:43:19.643862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.643792    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-84d7d4f6df-zdhj5" event={"ID":"eb4efd7c-1575-48aa-b5a8-22ecdbd87729","Type":"ContainerDied","Data":"071607b7254ddc106d0f04fa23af2d2d5aa447629216379d95b332029d624ef7"}
Jul 28 10:43:19.643862 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.643822    2578 scope.go:117] "RemoveContainer" containerID="e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e"
Jul 28 10:43:19.651964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.651927    2578 scope.go:117] "RemoveContainer" containerID="e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e"
Jul 28 10:43:19.652485 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:43:19.652453    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e\": container with ID starting with e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e not found: ID does not exist" containerID="e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e"
Jul 28 10:43:19.652590 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.652495    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e"} err="failed to get container status \"e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e\": rpc error: code = NotFound desc = could not find container \"e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e\": container with ID starting with e54e7b6cf643c44899f9615234e50bc131f641c3d5aa5ba2fcec0f3fbf1ffd6e not found: ID does not exist"
Jul 28 10:43:19.737401 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.737361    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsf6r\" (UniqueName: \"kubernetes.io/projected/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-kube-api-access-zsf6r\") pod \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") "
Jul 28 10:43:19.737691 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.737456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-webhook-cert\") pod \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\" (UID: \"eb4efd7c-1575-48aa-b5a8-22ecdbd87729\") "
Jul 28 10:43:19.739714 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.739673    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-kube-api-access-zsf6r" (OuterVolumeSpecName: "kube-api-access-zsf6r") pod "eb4efd7c-1575-48aa-b5a8-22ecdbd87729" (UID: "eb4efd7c-1575-48aa-b5a8-22ecdbd87729"). InnerVolumeSpecName "kube-api-access-zsf6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:43:19.739831 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.739683    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "eb4efd7c-1575-48aa-b5a8-22ecdbd87729" (UID: "eb4efd7c-1575-48aa-b5a8-22ecdbd87729"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:43:19.838902 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.838860    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsf6r\" (UniqueName: \"kubernetes.io/projected/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-kube-api-access-zsf6r\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:43:19.838902 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.838897    2578 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb4efd7c-1575-48aa-b5a8-22ecdbd87729-webhook-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:43:19.966261 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.966216    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-zdhj5"]
Jul 28 10:43:19.968737 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:19.968703    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-84d7d4f6df-zdhj5"]
Jul 28 10:43:21.493597 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:21.493563    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb4efd7c-1575-48aa-b5a8-22ecdbd87729" path="/var/lib/kubelet/pods/eb4efd7c-1575-48aa-b5a8-22ecdbd87729/volumes"
Jul 28 10:43:29.861071 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:29.861046    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:43:29.863292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:43:29.863255    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:45:00.137716 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.137674    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"]
Jul 28 10:45:00.140088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.137994    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb4efd7c-1575-48aa-b5a8-22ecdbd87729" containerName="manager"
Jul 28 10:45:00.140088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.138004    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb4efd7c-1575-48aa-b5a8-22ecdbd87729" containerName="manager"
Jul 28 10:45:00.140088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.138068    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb4efd7c-1575-48aa-b5a8-22ecdbd87729" containerName="manager"
Jul 28 10:45:00.140971 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.140955    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"
Jul 28 10:45:00.143449 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.143425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 10:45:00.148575 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.148549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"]
Jul 28 10:45:00.151748 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.151724    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"
Jul 28 10:45:00.287527 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:00.287502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"]
Jul 28 10:45:00.289679 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:45:00.289642    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac919f82_e99d_406b_acbd_c95a4beaa4bf.slice/crio-0c5a432529378e95dd14f37654dccad5bb8c1915979e264fec94515eccd57b5a WatchSource:0}: Error finding container 0c5a432529378e95dd14f37654dccad5bb8c1915979e264fec94515eccd57b5a: Status 404 returned error can't find the container with id 0c5a432529378e95dd14f37654dccad5bb8c1915979e264fec94515eccd57b5a
Jul 28 10:45:01.091673 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:01.091632    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9" event={"ID":"ac919f82-e99d-406b-acbd-c95a4beaa4bf","Type":"ContainerStarted","Data":"0c5a432529378e95dd14f37654dccad5bb8c1915979e264fec94515eccd57b5a"}
Jul 28 10:45:03.102914 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:03.102822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9" event={"ID":"ac919f82-e99d-406b-acbd-c95a4beaa4bf","Type":"ContainerStarted","Data":"430185caa7724d9bac8712ac2fdd2ae9d362314b9e626f9543a051831f42d483"}
Jul 28 10:45:03.117750 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:03.117694    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9" podStartSLOduration=1.013008211 podStartE2EDuration="3.117678291s" totalImagesPullingTime="2.10467008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:45:03.117231621 +0000 UTC m=+1294.159323785" watchObservedRunningTime="2026-07-28 10:45:03.117678291 +0000 UTC m=+1294.159770443"
Jul 28 10:45:04.109735 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:04.109690    2578 generic.go:356] "Generic (PLEG): container finished" podID="ac919f82-e99d-406b-acbd-c95a4beaa4bf" containerID="430185caa7724d9bac8712ac2fdd2ae9d362314b9e626f9543a051831f42d483" exitCode=0
Jul 28 10:45:04.110255 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:04.109757    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9" event={"ID":"ac919f82-e99d-406b-acbd-c95a4beaa4bf","Type":"ContainerDied","Data":"430185caa7724d9bac8712ac2fdd2ae9d362314b9e626f9543a051831f42d483"}
Jul 28 10:45:05.244840 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:05.244810    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"
Jul 28 10:45:06.119591 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:06.119554    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9" event={"ID":"ac919f82-e99d-406b-acbd-c95a4beaa4bf","Type":"ContainerDied","Data":"0c5a432529378e95dd14f37654dccad5bb8c1915979e264fec94515eccd57b5a"}
Jul 28 10:45:06.119591 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:06.119581    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753925-jqgk9"
Jul 28 10:45:06.119804 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:45:06.119612    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c5a432529378e95dd14f37654dccad5bb8c1915979e264fec94515eccd57b5a"
Jul 28 10:48:29.889496 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:48:29.889464    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:48:29.892174 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:48:29.892151    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:52:18.723476 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.723397    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"]
Jul 28 10:52:18.726081 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.723723    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac919f82-e99d-406b-acbd-c95a4beaa4bf" containerName="cleanup"
Jul 28 10:52:18.726081 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.723733    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac919f82-e99d-406b-acbd-c95a4beaa4bf" containerName="cleanup"
Jul 28 10:52:18.726081 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.723792    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac919f82-e99d-406b-acbd-c95a4beaa4bf" containerName="cleanup"
Jul 28 10:52:18.727069 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.727050    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.732208 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.732181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6215f6af-kserve-self-signed-certs\""
Jul 28 10:52:18.763761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.763724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"]
Jul 28 10:52:18.862935 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.862885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz528\" (UniqueName: \"kubernetes.io/projected/fb63882f-6aac-4f58-89ae-a31484a886d8-kube-api-access-tz528\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.862935 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.862940    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-dshm\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.863151 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.862961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb63882f-6aac-4f58-89ae-a31484a886d8-tls-certs\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.863151 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.863019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-model-cache\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.863151 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.863077    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-home\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.863151 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.863129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-tmp-dir\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.963926 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.963890    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-dshm\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.963926 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.963926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb63882f-6aac-4f58-89ae-a31484a886d8-tls-certs\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.963948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-model-cache\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.963967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-home\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.964012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-tmp-dir\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964173 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.964056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz528\" (UniqueName: \"kubernetes.io/projected/fb63882f-6aac-4f58-89ae-a31484a886d8-kube-api-access-tz528\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964425 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.964390    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-model-cache\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964473 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.964444    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-home\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.964473 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.964457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-tmp-dir\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.966290 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.966247    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-dshm\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.966575 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.966556    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb63882f-6aac-4f58-89ae-a31484a886d8-tls-certs\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:18.976864 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:18.976798    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz528\" (UniqueName: \"kubernetes.io/projected/fb63882f-6aac-4f58-89ae-a31484a886d8-kube-api-access-tz528\") pod \"e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") " pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:19.036436 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:19.036400    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:19.178892 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:19.178864    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"]
Jul 28 10:52:19.180467 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:52:19.180436    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb63882f_6aac_4f58_89ae_a31484a886d8.slice/crio-e56c4df18fa2fcdd8aae5f76b679480a1c5542c755218cd5f957319990f736a7 WatchSource:0}: Error finding container e56c4df18fa2fcdd8aae5f76b679480a1c5542c755218cd5f957319990f736a7: Status 404 returned error can't find the container with id e56c4df18fa2fcdd8aae5f76b679480a1c5542c755218cd5f957319990f736a7
Jul 28 10:52:20.047927 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:20.047893    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8" event={"ID":"fb63882f-6aac-4f58-89ae-a31484a886d8","Type":"ContainerStarted","Data":"856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b"}
Jul 28 10:52:20.047927 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:20.047935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8" event={"ID":"fb63882f-6aac-4f58-89ae-a31484a886d8","Type":"ContainerStarted","Data":"e56c4df18fa2fcdd8aae5f76b679480a1c5542c755218cd5f957319990f736a7"}
Jul 28 10:52:20.068241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:20.068188    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8" podStartSLOduration=2.068172098 podStartE2EDuration="2.068172098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:52:20.067847263 +0000 UTC m=+1731.109939419" watchObservedRunningTime="2026-07-28 10:52:20.068172098 +0000 UTC m=+1731.110264250"
Jul 28 10:52:29.036722 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:29.036679    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:29.037164 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:29.036765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:29.049916 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:29.049892    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:52:29.102821 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:52:29.102785    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:53:29.914891 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:29.914859    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:53:29.919750 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:29.919729    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:53:39.117381 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:39.117339    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"]
Jul 28 10:53:54.137132 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.137071    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8" podUID="fb63882f-6aac-4f58-89ae-a31484a886d8" containerName="main" containerID="cri-o://856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b" gracePeriod=45
Jul 28 10:53:54.379026 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.378997    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:53:54.466822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.466786    2578 generic.go:356] "Generic (PLEG): container finished" podID="fb63882f-6aac-4f58-89ae-a31484a886d8" containerID="856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b" exitCode=0
Jul 28 10:53:54.466822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.466825    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"
Jul 28 10:53:54.467039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.466871    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8" event={"ID":"fb63882f-6aac-4f58-89ae-a31484a886d8","Type":"ContainerDied","Data":"856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b"}
Jul 28 10:53:54.467039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.466917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8" event={"ID":"fb63882f-6aac-4f58-89ae-a31484a886d8","Type":"ContainerDied","Data":"e56c4df18fa2fcdd8aae5f76b679480a1c5542c755218cd5f957319990f736a7"}
Jul 28 10:53:54.467039 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.466937    2578 scope.go:117] "RemoveContainer" containerID="856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b"
Jul 28 10:53:54.475457 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.475432    2578 scope.go:117] "RemoveContainer" containerID="856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b"
Jul 28 10:53:54.475750 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:53:54.475732    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b\": container with ID starting with 856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b not found: ID does not exist" containerID="856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b"
Jul 28 10:53:54.475796 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.475762    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b"} err="failed to get container status \"856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b\": rpc error: code = NotFound desc = could not find container \"856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b\": container with ID starting with 856a3fdf52dd87e1b0cdcf8b621852757a8a5ce9083418c564e00c9dbac6ac8b not found: ID does not exist"
Jul 28 10:53:54.490431 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-model-cache\") pod \"fb63882f-6aac-4f58-89ae-a31484a886d8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") "
Jul 28 10:53:54.490616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490446    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-tmp-dir\") pod \"fb63882f-6aac-4f58-89ae-a31484a886d8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") "
Jul 28 10:53:54.490616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490480    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-dshm\") pod \"fb63882f-6aac-4f58-89ae-a31484a886d8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") "
Jul 28 10:53:54.490616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490510    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb63882f-6aac-4f58-89ae-a31484a886d8-tls-certs\") pod \"fb63882f-6aac-4f58-89ae-a31484a886d8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") "
Jul 28 10:53:54.490616 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490581    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tz528\" (UniqueName: \"kubernetes.io/projected/fb63882f-6aac-4f58-89ae-a31484a886d8-kube-api-access-tz528\") pod \"fb63882f-6aac-4f58-89ae-a31484a886d8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") "
Jul 28 10:53:54.490795 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490631    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-home\") pod \"fb63882f-6aac-4f58-89ae-a31484a886d8\" (UID: \"fb63882f-6aac-4f58-89ae-a31484a886d8\") "
Jul 28 10:53:54.491020 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.490956    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fb63882f-6aac-4f58-89ae-a31484a886d8" (UID: "fb63882f-6aac-4f58-89ae-a31484a886d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:53:54.491153 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.491016    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-model-cache" (OuterVolumeSpecName: "model-cache") pod "fb63882f-6aac-4f58-89ae-a31484a886d8" (UID: "fb63882f-6aac-4f58-89ae-a31484a886d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:53:54.491378 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.491228    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:53:54.491378 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.491253    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:53:54.491497 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.491405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-home" (OuterVolumeSpecName: "home") pod "fb63882f-6aac-4f58-89ae-a31484a886d8" (UID: "fb63882f-6aac-4f58-89ae-a31484a886d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:53:54.498241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.498176    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb63882f-6aac-4f58-89ae-a31484a886d8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fb63882f-6aac-4f58-89ae-a31484a886d8" (UID: "fb63882f-6aac-4f58-89ae-a31484a886d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:53:54.498241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.498183    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-dshm" (OuterVolumeSpecName: "dshm") pod "fb63882f-6aac-4f58-89ae-a31484a886d8" (UID: "fb63882f-6aac-4f58-89ae-a31484a886d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:53:54.498241 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.498188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb63882f-6aac-4f58-89ae-a31484a886d8-kube-api-access-tz528" (OuterVolumeSpecName: "kube-api-access-tz528") pod "fb63882f-6aac-4f58-89ae-a31484a886d8" (UID: "fb63882f-6aac-4f58-89ae-a31484a886d8"). InnerVolumeSpecName "kube-api-access-tz528". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:53:54.592355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.592316    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:53:54.592355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.592354    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb63882f-6aac-4f58-89ae-a31484a886d8-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:53:54.592355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.592363    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb63882f-6aac-4f58-89ae-a31484a886d8-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:53:54.592585 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.592376    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz528\" (UniqueName: \"kubernetes.io/projected/fb63882f-6aac-4f58-89ae-a31484a886d8-kube-api-access-tz528\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:53:54.788296 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.788250    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"]
Jul 28 10:53:54.791478 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:54.791453    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6215f6af-kserve-d5bcfb5f5-9whn8"]
Jul 28 10:53:55.494139 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:53:55.494101    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb63882f-6aac-4f58-89ae-a31484a886d8" path="/var/lib/kubelet/pods/fb63882f-6aac-4f58-89ae-a31484a886d8/volumes"
Jul 28 10:57:36.528984 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.528941    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"]
Jul 28 10:57:36.531321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.529282    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb63882f-6aac-4f58-89ae-a31484a886d8" containerName="main"
Jul 28 10:57:36.531321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.529294    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb63882f-6aac-4f58-89ae-a31484a886d8" containerName="main"
Jul 28 10:57:36.531321 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.529356    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb63882f-6aac-4f58-89ae-a31484a886d8" containerName="main"
Jul 28 10:57:36.532164 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.532148    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.542761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.542736    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3eefdb25-dockercfg-lqn49\""
Jul 28 10:57:36.543671 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.543651    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3eefdb25\""
Jul 28 10:57:36.556159 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.556118    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"]
Jul 28 10:57:36.625884 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.625834    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36aa9a-f324-4129-934c-469c54614a27-plugins-config-volume\") pod \"payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") " pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.626086 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.625895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdcww\" (UniqueName: \"kubernetes.io/projected/3f36aa9a-f324-4129-934c-469c54614a27-kube-api-access-mdcww\") pod \"payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") " pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.727221 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.727179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36aa9a-f324-4129-934c-469c54614a27-plugins-config-volume\") pod \"payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") " pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.727440 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.727249    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdcww\" (UniqueName: \"kubernetes.io/projected/3f36aa9a-f324-4129-934c-469c54614a27-kube-api-access-mdcww\") pod \"payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") " pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.727990 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.727964    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36aa9a-f324-4129-934c-469c54614a27-plugins-config-volume\") pod \"payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") " pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.735661 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.735637    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdcww\" (UniqueName: \"kubernetes.io/projected/3f36aa9a-f324-4129-934c-469c54614a27-kube-api-access-mdcww\") pod \"payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") " pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.842203 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.842102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:36.990513 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.990456    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"]
Jul 28 10:57:36.993412 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:57:36.993383    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f36aa9a_f324_4129_934c_469c54614a27.slice/crio-91d4c4386c45d6a71073ae2e148dbe5fb892caf001e70e2342066672555d7660 WatchSource:0}: Error finding container 91d4c4386c45d6a71073ae2e148dbe5fb892caf001e70e2342066672555d7660: Status 404 returned error can't find the container with id 91d4c4386c45d6a71073ae2e148dbe5fb892caf001e70e2342066672555d7660
Jul 28 10:57:36.995329 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:36.995261    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:57:37.451137 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:37.451096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp" event={"ID":"3f36aa9a-f324-4129-934c-469c54614a27","Type":"ContainerStarted","Data":"91d4c4386c45d6a71073ae2e148dbe5fb892caf001e70e2342066672555d7660"}
Jul 28 10:57:39.463581 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:39.463548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp" event={"ID":"3f36aa9a-f324-4129-934c-469c54614a27","Type":"ContainerStarted","Data":"3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e"}
Jul 28 10:57:39.463950 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:39.463627    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:57:39.482537 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:39.482479    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp" podStartSLOduration=1.994506528 podStartE2EDuration="3.482461384s" totalImagesPullingTime="1.487954856s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:57:39.48116586 +0000 UTC m=+2050.523258013" watchObservedRunningTime="2026-07-28 10:57:39.482461384 +0000 UTC m=+2050.524553537"
Jul 28 10:57:50.471017 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:57:50.470930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:58:01.236589 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.236551    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"]
Jul 28 10:58:01.239894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.239876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:01.242535 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.242514    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 10:58:01.242660 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.242514    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 10:58:01.248509 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.248474    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"]
Jul 28 10:58:01.336195 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.336156    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a84ea1-3355-495a-a4c6-616ad7805146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m\" (UID: \"e9a84ea1-3355-495a-a4c6-616ad7805146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:01.437340 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.437258    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a84ea1-3355-495a-a4c6-616ad7805146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m\" (UID: \"e9a84ea1-3355-495a-a4c6-616ad7805146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:01.437904 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.437884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a84ea1-3355-495a-a4c6-616ad7805146-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m\" (UID: \"e9a84ea1-3355-495a-a4c6-616ad7805146\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:01.552297 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.552158    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:01.689516 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:01.689478    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"]
Jul 28 10:58:01.693812 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:58:01.693781    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9a84ea1_3355_495a_a4c6_616ad7805146.slice/crio-1db6309ae2e4c3588e5f9c3e8572000ca4f2e38dd7959113eb081bad649f4d94 WatchSource:0}: Error finding container 1db6309ae2e4c3588e5f9c3e8572000ca4f2e38dd7959113eb081bad649f4d94: Status 404 returned error can't find the container with id 1db6309ae2e4c3588e5f9c3e8572000ca4f2e38dd7959113eb081bad649f4d94
Jul 28 10:58:02.570162 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:02.570129    2578 generic.go:356] "Generic (PLEG): container finished" podID="e9a84ea1-3355-495a-a4c6-616ad7805146" containerID="3172106ecf725007ad0cc373a987cf7097c66212ebfa7dcdcb9324550ec2edf1" exitCode=0
Jul 28 10:58:02.570579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:02.570212    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m" event={"ID":"e9a84ea1-3355-495a-a4c6-616ad7805146","Type":"ContainerDied","Data":"3172106ecf725007ad0cc373a987cf7097c66212ebfa7dcdcb9324550ec2edf1"}
Jul 28 10:58:02.570579 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:02.570315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m" event={"ID":"e9a84ea1-3355-495a-a4c6-616ad7805146","Type":"ContainerStarted","Data":"1db6309ae2e4c3588e5f9c3e8572000ca4f2e38dd7959113eb081bad649f4d94"}
Jul 28 10:58:03.717225 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:03.717200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:03.760639 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:03.760603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a84ea1-3355-495a-a4c6-616ad7805146-maas-api-service-ca\") pod \"e9a84ea1-3355-495a-a4c6-616ad7805146\" (UID: \"e9a84ea1-3355-495a-a4c6-616ad7805146\") "
Jul 28 10:58:03.760965 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:03.760939    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9a84ea1-3355-495a-a4c6-616ad7805146-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9a84ea1-3355-495a-a4c6-616ad7805146" (UID: "e9a84ea1-3355-495a-a4c6-616ad7805146"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:58:03.862242 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:03.862140    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9a84ea1-3355-495a-a4c6-616ad7805146-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:58:04.580858 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:04.580823    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"
Jul 28 10:58:04.581034 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:04.580855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m" event={"ID":"e9a84ea1-3355-495a-a4c6-616ad7805146","Type":"ContainerDied","Data":"1db6309ae2e4c3588e5f9c3e8572000ca4f2e38dd7959113eb081bad649f4d94"}
Jul 28 10:58:04.581034 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:04.580905    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1db6309ae2e4c3588e5f9c3e8572000ca4f2e38dd7959113eb081bad649f4d94"
Jul 28 10:58:11.682013 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:11.681979    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"]
Jul 28 10:58:11.682493 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:11.682175    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp" podUID="3f36aa9a-f324-4129-934c-469c54614a27" containerName="payload-processing" containerID="cri-o://3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e" gracePeriod=30
Jul 28 10:58:11.930456 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:11.930426    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:58:12.031498 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.031399    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdcww\" (UniqueName: \"kubernetes.io/projected/3f36aa9a-f324-4129-934c-469c54614a27-kube-api-access-mdcww\") pod \"3f36aa9a-f324-4129-934c-469c54614a27\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") "
Jul 28 10:58:12.031672 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.031522    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36aa9a-f324-4129-934c-469c54614a27-plugins-config-volume\") pod \"3f36aa9a-f324-4129-934c-469c54614a27\" (UID: \"3f36aa9a-f324-4129-934c-469c54614a27\") "
Jul 28 10:58:12.031932 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.031906    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f36aa9a-f324-4129-934c-469c54614a27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f36aa9a-f324-4129-934c-469c54614a27" (UID: "3f36aa9a-f324-4129-934c-469c54614a27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:58:12.033794 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.033770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f36aa9a-f324-4129-934c-469c54614a27-kube-api-access-mdcww" (OuterVolumeSpecName: "kube-api-access-mdcww") pod "3f36aa9a-f324-4129-934c-469c54614a27" (UID: "3f36aa9a-f324-4129-934c-469c54614a27"). InnerVolumeSpecName "kube-api-access-mdcww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:58:12.132496 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.132460    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f36aa9a-f324-4129-934c-469c54614a27-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:58:12.132496 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.132495    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdcww\" (UniqueName: \"kubernetes.io/projected/3f36aa9a-f324-4129-934c-469c54614a27-kube-api-access-mdcww\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:58:12.620509 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.620477    2578 generic.go:356] "Generic (PLEG): container finished" podID="3f36aa9a-f324-4129-934c-469c54614a27" containerID="3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e" exitCode=0
Jul 28 10:58:12.620689 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.620521    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"
Jul 28 10:58:12.620689 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.620537    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp" event={"ID":"3f36aa9a-f324-4129-934c-469c54614a27","Type":"ContainerDied","Data":"3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e"}
Jul 28 10:58:12.620689 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.620581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp" event={"ID":"3f36aa9a-f324-4129-934c-469c54614a27","Type":"ContainerDied","Data":"91d4c4386c45d6a71073ae2e148dbe5fb892caf001e70e2342066672555d7660"}
Jul 28 10:58:12.620689 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.620602    2578 scope.go:117] "RemoveContainer" containerID="3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e"
Jul 28 10:58:12.631994 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.631973    2578 scope.go:117] "RemoveContainer" containerID="3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e"
Jul 28 10:58:12.632337 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:58:12.632314    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e\": container with ID starting with 3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e not found: ID does not exist" containerID="3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e"
Jul 28 10:58:12.632399 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.632347    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e"} err="failed to get container status \"3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e\": rpc error: code = NotFound desc = could not find container \"3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e\": container with ID starting with 3fc280a3e0ea973ee1715cd4b8dcbfff0421c252b0abab40e2b4e76fdbab253e not found: ID does not exist"
Jul 28 10:58:12.640741 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.640707    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"]
Jul 28 10:58:12.645263 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:12.645234    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3eefdb25-5c7c86b876-mn6qp"]
Jul 28 10:58:13.498125 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:13.498088    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f36aa9a-f324-4129-934c-469c54614a27" path="/var/lib/kubelet/pods/3f36aa9a-f324-4129-934c-469c54614a27/volumes"
Jul 28 10:58:16.304286 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:16.304232    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"]
Jul 28 10:58:16.306936 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:16.306909    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3eefdb25-hl22m"]
Jul 28 10:58:17.493777 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.493732    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a84ea1-3355-495a-a4c6-616ad7805146" path="/var/lib/kubelet/pods/e9a84ea1-3355-495a-a4c6-616ad7805146/volumes"
Jul 28 10:58:17.682464 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682426    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"]
Jul 28 10:58:17.682776 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682761    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a84ea1-3355-495a-a4c6-616ad7805146" containerName="revoke-keys"
Jul 28 10:58:17.682820 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682780    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a84ea1-3355-495a-a4c6-616ad7805146" containerName="revoke-keys"
Jul 28 10:58:17.682820 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f36aa9a-f324-4129-934c-469c54614a27" containerName="payload-processing"
Jul 28 10:58:17.682820 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682806    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f36aa9a-f324-4129-934c-469c54614a27" containerName="payload-processing"
Jul 28 10:58:17.682910 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682867    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f36aa9a-f324-4129-934c-469c54614a27" containerName="payload-processing"
Jul 28 10:58:17.682910 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.682877    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a84ea1-3355-495a-a4c6-616ad7805146" containerName="revoke-keys"
Jul 28 10:58:17.688554 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.688534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.696675 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.696658    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fde3f8de-gw-openshift-default-dockercfg-m225c\""
Jul 28 10:58:17.696782 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.696725    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 10:58:17.718682 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.718652    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"]
Jul 28 10:58:17.777814 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-istio-token\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.777814 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2287e14a-2203-423f-b6a4-babc60775fd0-istiod-ca-cert\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.777814 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777790    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-credential-socket\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.778056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk5g6\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-kube-api-access-zk5g6\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.778056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-certs\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.778056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777892    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2287e14a-2203-423f-b6a4-babc60775fd0-istio-podinfo\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.778056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-socket\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.778056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-data\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.778056 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.777979    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-envoy\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk5g6\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-kube-api-access-zk5g6\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879127 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-certs\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2287e14a-2203-423f-b6a4-babc60775fd0-istio-podinfo\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-socket\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879221    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-data\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-envoy\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-istio-token\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879356    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2287e14a-2203-423f-b6a4-babc60775fd0-istiod-ca-cert\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879409 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879400    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-credential-socket\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879601    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-certs\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879649    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-socket\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-data\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.879917 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.879784    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-credential-socket\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.880177 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.880162    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2287e14a-2203-423f-b6a4-babc60775fd0-istiod-ca-cert\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.881876 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.881855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-envoy\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.882163 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.882146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2287e14a-2203-423f-b6a4-babc60775fd0-istio-podinfo\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.888959 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.888937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-istio-token\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.889126 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.889105    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk5g6\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-kube-api-access-zk5g6\") pod \"e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") " pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:17.998674 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:17.998631    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:18.138471 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:18.138430    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"]
Jul 28 10:58:18.140666 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:58:18.140637    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2287e14a_2203_423f_b6a4_babc60775fd0.slice/crio-15772efe2209b8c58f963c506e57a153d9a8781f5d368dc15a216baa2cd09fb8 WatchSource:0}: Error finding container 15772efe2209b8c58f963c506e57a153d9a8781f5d368dc15a216baa2cd09fb8: Status 404 returned error can't find the container with id 15772efe2209b8c58f963c506e57a153d9a8781f5d368dc15a216baa2cd09fb8
Jul 28 10:58:18.649231 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:18.649194    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" event={"ID":"2287e14a-2203-423f-b6a4-babc60775fd0","Type":"ContainerStarted","Data":"15772efe2209b8c58f963c506e57a153d9a8781f5d368dc15a216baa2cd09fb8"}
Jul 28 10:58:29.946107 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:29.946075    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:58:29.952433 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:29.952407    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 10:58:31.343156 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:31.342419    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 10:58:31.343156 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:31.342524    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 10:58:31.343156 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:31.342579    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 10:58:31.709406 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:31.709368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" event={"ID":"2287e14a-2203-423f-b6a4-babc60775fd0","Type":"ContainerStarted","Data":"5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"}
Jul 28 10:58:31.732645 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:31.732589    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" podStartSLOduration=1.5332357079999999 podStartE2EDuration="14.732570947s" totalImagesPullingTime="13.199335239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:58:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:58:31.731215099 +0000 UTC m=+2102.773307252" watchObservedRunningTime="2026-07-28 10:58:31.732570947 +0000 UTC m=+2102.774663100"
Jul 28 10:58:31.999227 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:31.999132    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:32.999280 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:32.999240    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:33.004292 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:33.004254    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:33.719593 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:33.719558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:34.003176 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.003095    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"]
Jul 28 10:58:34.006599 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.006572    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.009103 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.009083    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fde3f8de-dockercfg-brz9g\""
Jul 28 10:58:34.009210 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.009105    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fde3f8de\""
Jul 28 10:58:34.015825 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.015798    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"]
Jul 28 10:58:34.131399 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.131353    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvm5v\" (UniqueName: \"kubernetes.io/projected/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-kube-api-access-nvm5v\") pod \"payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") " pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.131588 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.131444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") " pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.232381 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.232331    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvm5v\" (UniqueName: \"kubernetes.io/projected/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-kube-api-access-nvm5v\") pod \"payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") " pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.232558 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.232427    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") " pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.233145 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.233125    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-plugins-config-volume\") pod \"payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") " pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.240515 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.240478    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvm5v\" (UniqueName: \"kubernetes.io/projected/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-kube-api-access-nvm5v\") pod \"payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") " pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.317998 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.317887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:34.456618 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.456587    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"]
Jul 28 10:58:34.458760 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:58:34.458732    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63ca9cf_5830_4d02_9e63_4afdd4b8a922.slice/crio-d168a7c4a6d6464428f32f1d1029eaa37c6436a8f779b131394b9f52dad0789f WatchSource:0}: Error finding container d168a7c4a6d6464428f32f1d1029eaa37c6436a8f779b131394b9f52dad0789f: Status 404 returned error can't find the container with id d168a7c4a6d6464428f32f1d1029eaa37c6436a8f779b131394b9f52dad0789f
Jul 28 10:58:34.725370 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.725332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2" event={"ID":"e63ca9cf-5830-4d02-9e63-4afdd4b8a922","Type":"ContainerStarted","Data":"86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb"}
Jul 28 10:58:34.725549 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:34.725394    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2" event={"ID":"e63ca9cf-5830-4d02-9e63-4afdd4b8a922","Type":"ContainerStarted","Data":"d168a7c4a6d6464428f32f1d1029eaa37c6436a8f779b131394b9f52dad0789f"}
Jul 28 10:58:35.730328 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:35.730299    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:45.780433 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.780337    2578 generic.go:356] "Generic (PLEG): container finished" podID="2287e14a-2203-423f-b6a4-babc60775fd0" containerID="5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0" exitCode=0
Jul 28 10:58:45.780433 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.780391    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" event={"ID":"2287e14a-2203-423f-b6a4-babc60775fd0","Type":"ContainerDied","Data":"5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"}
Jul 28 10:58:45.780851 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.780771    2578 scope.go:117] "RemoveContainer" containerID="5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"
Jul 28 10:58:45.781758 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.781727    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 10:58:45.781815 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.781797    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 10:58:45.781851 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.781828    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 10:58:45.801472 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:45.801410    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2" podStartSLOduration=12.801386371 podStartE2EDuration="12.801386371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:58:34.741683692 +0000 UTC m=+2105.783775846" watchObservedRunningTime="2026-07-28 10:58:45.801386371 +0000 UTC m=+2116.843478529"
Jul 28 10:58:46.738338 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:46.738304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:58:46.786974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:46.786934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" event={"ID":"2287e14a-2203-423f-b6a4-babc60775fd0","Type":"ContainerStarted","Data":"e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5"}
Jul 28 10:58:46.999716 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:46.999628    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:47.000879 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:47.000853    2578 patch_prober.go:28] interesting pod/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 28 10:58:47.001208 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:47.000928    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 28 10:58:47.999137 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:47.999093    2578 patch_prober.go:28] interesting pod/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 28 10:58:47.999588 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:47.999154    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 28 10:58:48.719160 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:48.719107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:48.999055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:48.998972    2578 patch_prober.go:28] interesting pod/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 28 10:58:48.999055 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:48.999036    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 28 10:58:49.999628 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:49.999585    2578 patch_prober.go:28] interesting pod/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 28 10:58:50.000031 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:49.999664    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 28 10:58:51.003316 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:51.003264    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:51.004188 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:51.004167    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:58:58.353178 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.353134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"]
Jul 28 10:58:58.362258 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.362225    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:58:58.363889 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.363859    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"]
Jul 28 10:58:58.364887 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.364861    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 10:58:58.365010 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.364894    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 10:58:58.438692 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.438649    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569b4bb-34f9-4e42-a7a6-c85c281ba87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6\" (UID: \"4569b4bb-34f9-4e42-a7a6-c85c281ba87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:58:58.539515 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.539478    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569b4bb-34f9-4e42-a7a6-c85c281ba87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6\" (UID: \"4569b4bb-34f9-4e42-a7a6-c85c281ba87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:58:58.540088 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.540067    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569b4bb-34f9-4e42-a7a6-c85c281ba87b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6\" (UID: \"4569b4bb-34f9-4e42-a7a6-c85c281ba87b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:58:58.675119 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.675085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:58:58.814333 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.814307    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"]
Jul 28 10:58:58.816573 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:58:58.816538    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4569b4bb_34f9_4e42_a7a6_c85c281ba87b.slice/crio-3d36323ef31c88b87c4cf5ade0a0924d2ac92637c593128b68a393e298792a49 WatchSource:0}: Error finding container 3d36323ef31c88b87c4cf5ade0a0924d2ac92637c593128b68a393e298792a49: Status 404 returned error can't find the container with id 3d36323ef31c88b87c4cf5ade0a0924d2ac92637c593128b68a393e298792a49
Jul 28 10:58:58.853208 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:58.853175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6" event={"ID":"4569b4bb-34f9-4e42-a7a6-c85c281ba87b","Type":"ContainerStarted","Data":"3d36323ef31c88b87c4cf5ade0a0924d2ac92637c593128b68a393e298792a49"}
Jul 28 10:58:59.859834 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:59.859738    2578 generic.go:356] "Generic (PLEG): container finished" podID="4569b4bb-34f9-4e42-a7a6-c85c281ba87b" containerID="c194c3068195852d631830a7e2074f1a5b11c89e8abe15af8bcfee856efc8bf5" exitCode=0
Jul 28 10:58:59.859834 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:58:59.859788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6" event={"ID":"4569b4bb-34f9-4e42-a7a6-c85c281ba87b","Type":"ContainerDied","Data":"c194c3068195852d631830a7e2074f1a5b11c89e8abe15af8bcfee856efc8bf5"}
Jul 28 10:59:00.996952 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:00.996918    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:59:01.062782 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:01.062739    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569b4bb-34f9-4e42-a7a6-c85c281ba87b-maas-api-service-ca\") pod \"4569b4bb-34f9-4e42-a7a6-c85c281ba87b\" (UID: \"4569b4bb-34f9-4e42-a7a6-c85c281ba87b\") "
Jul 28 10:59:01.063157 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:01.063134    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4569b4bb-34f9-4e42-a7a6-c85c281ba87b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4569b4bb-34f9-4e42-a7a6-c85c281ba87b" (UID: "4569b4bb-34f9-4e42-a7a6-c85c281ba87b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:01.163972 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:01.163885    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4569b4bb-34f9-4e42-a7a6-c85c281ba87b-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:01.870179 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:01.870090    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"
Jul 28 10:59:01.870179 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:01.870104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6" event={"ID":"4569b4bb-34f9-4e42-a7a6-c85c281ba87b","Type":"ContainerDied","Data":"3d36323ef31c88b87c4cf5ade0a0924d2ac92637c593128b68a393e298792a49"}
Jul 28 10:59:01.870179 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:01.870150    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d36323ef31c88b87c4cf5ade0a0924d2ac92637c593128b68a393e298792a49"
Jul 28 10:59:08.935213 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:08.935173    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"]
Jul 28 10:59:08.935655 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:08.935423    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2" podUID="e63ca9cf-5830-4d02-9e63-4afdd4b8a922" containerName="payload-processing" containerID="cri-o://86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb" gracePeriod=30
Jul 28 10:59:09.197522 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.197449    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:59:09.331580 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.331538    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-plugins-config-volume\") pod \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") "
Jul 28 10:59:09.331761 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.331598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvm5v\" (UniqueName: \"kubernetes.io/projected/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-kube-api-access-nvm5v\") pod \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\" (UID: \"e63ca9cf-5830-4d02-9e63-4afdd4b8a922\") "
Jul 28 10:59:09.332051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.332022    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e63ca9cf-5830-4d02-9e63-4afdd4b8a922" (UID: "e63ca9cf-5830-4d02-9e63-4afdd4b8a922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:09.333939 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.333914    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-kube-api-access-nvm5v" (OuterVolumeSpecName: "kube-api-access-nvm5v") pod "e63ca9cf-5830-4d02-9e63-4afdd4b8a922" (UID: "e63ca9cf-5830-4d02-9e63-4afdd4b8a922"). InnerVolumeSpecName "kube-api-access-nvm5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:59:09.432701 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.432659    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:09.432701 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.432694    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvm5v\" (UniqueName: \"kubernetes.io/projected/e63ca9cf-5830-4d02-9e63-4afdd4b8a922-kube-api-access-nvm5v\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:09.910767 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.910675    2578 generic.go:356] "Generic (PLEG): container finished" podID="e63ca9cf-5830-4d02-9e63-4afdd4b8a922" containerID="86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb" exitCode=0
Jul 28 10:59:09.910767 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.910738    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"
Jul 28 10:59:09.910950 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.910771    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2" event={"ID":"e63ca9cf-5830-4d02-9e63-4afdd4b8a922","Type":"ContainerDied","Data":"86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb"}
Jul 28 10:59:09.910950 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.910827    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2" event={"ID":"e63ca9cf-5830-4d02-9e63-4afdd4b8a922","Type":"ContainerDied","Data":"d168a7c4a6d6464428f32f1d1029eaa37c6436a8f779b131394b9f52dad0789f"}
Jul 28 10:59:09.910950 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.910852    2578 scope.go:117] "RemoveContainer" containerID="86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb"
Jul 28 10:59:09.919457 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.919432    2578 scope.go:117] "RemoveContainer" containerID="86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb"
Jul 28 10:59:09.920688 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:59:09.920653    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb\": container with ID starting with 86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb not found: ID does not exist" containerID="86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb"
Jul 28 10:59:09.920832 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.920697    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb"} err="failed to get container status \"86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb\": rpc error: code = NotFound desc = could not find container \"86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb\": container with ID starting with 86738d143aa95f85bf9cb6977012289a3bfd3e573c9857d2cf120d924130f0eb not found: ID does not exist"
Jul 28 10:59:09.932790 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.932761    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"]
Jul 28 10:59:09.938774 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:09.938740    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-fde3f8de-5fd594bd7c-wdsf2"]
Jul 28 10:59:11.494833 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:11.494795    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e63ca9cf-5830-4d02-9e63-4afdd4b8a922" path="/var/lib/kubelet/pods/e63ca9cf-5830-4d02-9e63-4afdd4b8a922/volumes"
Jul 28 10:59:13.431964 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:13.431920    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"]
Jul 28 10:59:13.434624 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:13.434596    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fde3f8de-cdlt6"]
Jul 28 10:59:13.493619 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:13.493590    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4569b4bb-34f9-4e42-a7a6-c85c281ba87b" path="/var/lib/kubelet/pods/4569b4bb-34f9-4e42-a7a6-c85c281ba87b/volumes"
Jul 28 10:59:13.578734 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:13.578695    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"]
Jul 28 10:59:13.578970 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:13.578945    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy" containerID="cri-o://e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5" gracePeriod=30
Jul 28 10:59:17.778476 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778438    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"]
Jul 28 10:59:17.778974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778778    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e63ca9cf-5830-4d02-9e63-4afdd4b8a922" containerName="payload-processing"
Jul 28 10:59:17.778974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778789    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e63ca9cf-5830-4d02-9e63-4afdd4b8a922" containerName="payload-processing"
Jul 28 10:59:17.778974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778800    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4569b4bb-34f9-4e42-a7a6-c85c281ba87b" containerName="revoke-keys"
Jul 28 10:59:17.778974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778805    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4569b4bb-34f9-4e42-a7a6-c85c281ba87b" containerName="revoke-keys"
Jul 28 10:59:17.778974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778870    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e63ca9cf-5830-4d02-9e63-4afdd4b8a922" containerName="payload-processing"
Jul 28 10:59:17.778974 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.778878    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4569b4bb-34f9-4e42-a7a6-c85c281ba87b" containerName="revoke-keys"
Jul 28 10:59:17.783319 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.783298    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:17.786505 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.786483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4217be61\""
Jul 28 10:59:17.786712 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.786700    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4217be61-dockercfg-4vv8n\""
Jul 28 10:59:17.793421 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.793396    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"]
Jul 28 10:59:17.905929 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.905890    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8009cd-1367-443c-9880-37cf6fc71c94-plugins-config-volume\") pod \"payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") " pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:17.905929 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:17.905939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9f8n\" (UniqueName: \"kubernetes.io/projected/9e8009cd-1367-443c-9880-37cf6fc71c94-kube-api-access-q9f8n\") pod \"payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") " pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.006880 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.006836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8009cd-1367-443c-9880-37cf6fc71c94-plugins-config-volume\") pod \"payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") " pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.006880 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.006883    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9f8n\" (UniqueName: \"kubernetes.io/projected/9e8009cd-1367-443c-9880-37cf6fc71c94-kube-api-access-q9f8n\") pod \"payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") " pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.007615 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.007593    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8009cd-1367-443c-9880-37cf6fc71c94-plugins-config-volume\") pod \"payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") " pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.015868 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.015840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9f8n\" (UniqueName: \"kubernetes.io/projected/9e8009cd-1367-443c-9880-37cf6fc71c94-kube-api-access-q9f8n\") pod \"payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") " pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.094801 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.094696    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.238219 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.238187    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"]
Jul 28 10:59:18.240992 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:59:18.240961    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8009cd_1367_443c_9880_37cf6fc71c94.slice/crio-f4905672c83129f338355e779a0357144b9d4f19d2eecaa73e846d8333235ac5 WatchSource:0}: Error finding container f4905672c83129f338355e779a0357144b9d4f19d2eecaa73e846d8333235ac5: Status 404 returned error can't find the container with id f4905672c83129f338355e779a0357144b9d4f19d2eecaa73e846d8333235ac5
Jul 28 10:59:18.820530 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.820500    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:59:18.914826 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.914724    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-credential-socket\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.914826 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.914781    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zk5g6\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-kube-api-access-zk5g6\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.914826 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.914821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-istio-token\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915093 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.914878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-envoy\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915093 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.914951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2287e14a-2203-423f-b6a4-babc60775fd0-istiod-ca-cert\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915093 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.914977    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2287e14a-2203-423f-b6a4-babc60775fd0-istio-podinfo\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915093 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915088    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-data\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915123    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-socket\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915331 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915151    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-certs\") pod \"2287e14a-2203-423f-b6a4-babc60775fd0\" (UID: \"2287e14a-2203-423f-b6a4-babc60775fd0\") "
Jul 28 10:59:18.915533 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915477    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:59:18.915674 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915609    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:59:18.915814 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2287e14a-2203-423f-b6a4-babc60775fd0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:18.915878 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.915795    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:18.916403 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.916373    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-data" (OuterVolumeSpecName: "istio-data") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:59:18.916522 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.916431    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:59:18.917477 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.917452    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-kube-api-access-zk5g6" (OuterVolumeSpecName: "kube-api-access-zk5g6") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "kube-api-access-zk5g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:59:18.917921 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.917904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 10:59:18.917994 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.917916    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-istio-token" (OuterVolumeSpecName: "istio-token") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:59:18.918568 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.918547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2287e14a-2203-423f-b6a4-babc60775fd0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2287e14a-2203-423f-b6a4-babc60775fd0" (UID: "2287e14a-2203-423f-b6a4-babc60775fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 10:59:18.953857 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.953818    2578 generic.go:356] "Generic (PLEG): container finished" podID="2287e14a-2203-423f-b6a4-babc60775fd0" containerID="e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5" exitCode=0
Jul 28 10:59:18.954051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.953877    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"
Jul 28 10:59:18.954051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.953906    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" event={"ID":"2287e14a-2203-423f-b6a4-babc60775fd0","Type":"ContainerDied","Data":"e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5"}
Jul 28 10:59:18.954051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.953959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw" event={"ID":"2287e14a-2203-423f-b6a4-babc60775fd0","Type":"ContainerDied","Data":"15772efe2209b8c58f963c506e57a153d9a8781f5d368dc15a216baa2cd09fb8"}
Jul 28 10:59:18.954051 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.953973    2578 scope.go:117] "RemoveContainer" containerID="e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5"
Jul 28 10:59:18.955779 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.955755    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78" event={"ID":"9e8009cd-1367-443c-9880-37cf6fc71c94","Type":"ContainerStarted","Data":"4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268"}
Jul 28 10:59:18.955895 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.955804    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78" event={"ID":"9e8009cd-1367-443c-9880-37cf6fc71c94","Type":"ContainerStarted","Data":"f4905672c83129f338355e779a0357144b9d4f19d2eecaa73e846d8333235ac5"}
Jul 28 10:59:18.955956 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.955903    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:18.963760 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.963738    2578 scope.go:117] "RemoveContainer" containerID="5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"
Jul 28 10:59:18.972199 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.972178    2578 scope.go:117] "RemoveContainer" containerID="e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5"
Jul 28 10:59:18.972549 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:59:18.972525    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5\": container with ID starting with e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5 not found: ID does not exist" containerID="e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5"
Jul 28 10:59:18.972623 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.972560    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5"} err="failed to get container status \"e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5\": rpc error: code = NotFound desc = could not find container \"e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5\": container with ID starting with e02b0f4231bc5f36155277b37077551b519f7dd8fa08f75e2326a34b0aa2fad5 not found: ID does not exist"
Jul 28 10:59:18.972623 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.972579    2578 scope.go:117] "RemoveContainer" containerID="5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"
Jul 28 10:59:18.972842 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:59:18.972821    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0\": container with ID starting with 5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0 not found: ID does not exist" containerID="5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"
Jul 28 10:59:18.972903 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.972851    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0"} err="failed to get container status \"5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0\": rpc error: code = NotFound desc = could not find container \"5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0\": container with ID starting with 5f3075dd7b2e2aa01cdf9db63ae7733605e19fec1365a441a439c2d22f6c4aa0 not found: ID does not exist"
Jul 28 10:59:18.975228 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.975182    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78" podStartSLOduration=1.975167366 podStartE2EDuration="1.975167366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:18.974712442 +0000 UTC m=+2150.016804595" watchObservedRunningTime="2026-07-28 10:59:18.975167366 +0000 UTC m=+2150.017259519"
Jul 28 10:59:18.989639 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.989602    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"]
Jul 28 10:59:18.993504 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:18.993473    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fde3f8de-gw-openshift-default-7d667fffdd-qdwvw"]
Jul 28 10:59:19.016820 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016785    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.016820 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016817    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2287e14a-2203-423f-b6a4-babc60775fd0-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.017025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016829    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2287e14a-2203-423f-b6a4-babc60775fd0-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.017025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016841    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.017025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016853    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.017025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016864    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2287e14a-2203-423f-b6a4-babc60775fd0-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.017025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016878    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk5g6\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-kube-api-access-zk5g6\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.017025 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.016891    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2287e14a-2203-423f-b6a4-babc60775fd0-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:19.494827 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:19.494795    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" path="/var/lib/kubelet/pods/2287e14a-2203-423f-b6a4-babc60775fd0/volumes"
Jul 28 10:59:29.963071 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:29.963037    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:42.993426 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993387    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"]
Jul 28 10:59:42.993822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993720    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy"
Jul 28 10:59:42.993822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993731    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy"
Jul 28 10:59:42.993822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993745    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy"
Jul 28 10:59:42.993822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993751    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy"
Jul 28 10:59:42.993822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993802    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy"
Jul 28 10:59:42.993822 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.993809    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2287e14a-2203-423f-b6a4-babc60775fd0" containerName="istio-proxy"
Jul 28 10:59:42.997513 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:42.997491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:43.000257 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.000229    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 10:59:43.000431 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.000311    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 10:59:43.003985 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.003951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"]
Jul 28 10:59:43.039426 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.039380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4217be61-5j27b\" (UID: \"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:43.140036 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.139991    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4217be61-5j27b\" (UID: \"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:43.140698 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.140673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4217be61-5j27b\" (UID: \"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:43.309983 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.309944    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:43.464700 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:43.464561    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"]
Jul 28 10:59:43.467193 ip-10-0-131-67 kubenswrapper[2578]: W0728 10:59:43.467162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4b8bb2_7c2b_4bc6_ba09_b9922c1d6dfa.slice/crio-281ecf6b8483cbba6c96c264f96ae7fe608150d87b3089aa1a1a898782b0f90a WatchSource:0}: Error finding container 281ecf6b8483cbba6c96c264f96ae7fe608150d87b3089aa1a1a898782b0f90a: Status 404 returned error can't find the container with id 281ecf6b8483cbba6c96c264f96ae7fe608150d87b3089aa1a1a898782b0f90a
Jul 28 10:59:44.081399 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:44.081361    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b" event={"ID":"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa","Type":"ContainerStarted","Data":"53527846a59c011b5f131c4002ef1c2b7dcefafabe3ad604f487d27bf574f51d"}
Jul 28 10:59:44.081788 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:44.081414    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b" event={"ID":"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa","Type":"ContainerStarted","Data":"281ecf6b8483cbba6c96c264f96ae7fe608150d87b3089aa1a1a898782b0f90a"}
Jul 28 10:59:44.095480 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:44.095432    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b" podStartSLOduration=2.095413223 podStartE2EDuration="2.095413223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:44.095400162 +0000 UTC m=+2175.137492318" watchObservedRunningTime="2026-07-28 10:59:44.095413223 +0000 UTC m=+2175.137505376"
Jul 28 10:59:45.087250 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:45.087210    2578 generic.go:356] "Generic (PLEG): container finished" podID="8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa" containerID="53527846a59c011b5f131c4002ef1c2b7dcefafabe3ad604f487d27bf574f51d" exitCode=0
Jul 28 10:59:45.087748 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:45.087300    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b" event={"ID":"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa","Type":"ContainerDied","Data":"53527846a59c011b5f131c4002ef1c2b7dcefafabe3ad604f487d27bf574f51d"}
Jul 28 10:59:46.226833 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:46.226806    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:46.275323 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:46.275254    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa-maas-api-service-ca\") pod \"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa\" (UID: \"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa\") "
Jul 28 10:59:46.275637 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:46.275610    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa" (UID: "8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:46.376751 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:46.376660    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:47.098619 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:47.098587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"
Jul 28 10:59:47.098795 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:47.098587    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b" event={"ID":"8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa","Type":"ContainerDied","Data":"281ecf6b8483cbba6c96c264f96ae7fe608150d87b3089aa1a1a898782b0f90a"}
Jul 28 10:59:47.098795 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:47.098713    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="281ecf6b8483cbba6c96c264f96ae7fe608150d87b3089aa1a1a898782b0f90a"
Jul 28 10:59:53.517453 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.517416    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"]
Jul 28 10:59:53.517920 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.517639    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78" podUID="9e8009cd-1367-443c-9880-37cf6fc71c94" containerName="payload-processing" containerID="cri-o://4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268" gracePeriod=30
Jul 28 10:59:53.769287 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.769204    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:53.843041 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.842997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8009cd-1367-443c-9880-37cf6fc71c94-plugins-config-volume\") pod \"9e8009cd-1367-443c-9880-37cf6fc71c94\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") "
Jul 28 10:59:53.843218 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.843079    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9f8n\" (UniqueName: \"kubernetes.io/projected/9e8009cd-1367-443c-9880-37cf6fc71c94-kube-api-access-q9f8n\") pod \"9e8009cd-1367-443c-9880-37cf6fc71c94\" (UID: \"9e8009cd-1367-443c-9880-37cf6fc71c94\") "
Jul 28 10:59:53.843474 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.843455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8009cd-1367-443c-9880-37cf6fc71c94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9e8009cd-1367-443c-9880-37cf6fc71c94" (UID: "9e8009cd-1367-443c-9880-37cf6fc71c94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:59:53.845355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.845332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e8009cd-1367-443c-9880-37cf6fc71c94-kube-api-access-q9f8n" (OuterVolumeSpecName: "kube-api-access-q9f8n") pod "9e8009cd-1367-443c-9880-37cf6fc71c94" (UID: "9e8009cd-1367-443c-9880-37cf6fc71c94"). InnerVolumeSpecName "kube-api-access-q9f8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:59:53.943894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.943857    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e8009cd-1367-443c-9880-37cf6fc71c94-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:53.943894 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:53.943886    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9f8n\" (UniqueName: \"kubernetes.io/projected/9e8009cd-1367-443c-9880-37cf6fc71c94-kube-api-access-q9f8n\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 10:59:54.130355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.130229    2578 generic.go:356] "Generic (PLEG): container finished" podID="9e8009cd-1367-443c-9880-37cf6fc71c94" containerID="4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268" exitCode=0
Jul 28 10:59:54.130355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.130283    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"
Jul 28 10:59:54.130355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.130291    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78" event={"ID":"9e8009cd-1367-443c-9880-37cf6fc71c94","Type":"ContainerDied","Data":"4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268"}
Jul 28 10:59:54.130355 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.130345    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78" event={"ID":"9e8009cd-1367-443c-9880-37cf6fc71c94","Type":"ContainerDied","Data":"f4905672c83129f338355e779a0357144b9d4f19d2eecaa73e846d8333235ac5"}
Jul 28 10:59:54.130683 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.130376    2578 scope.go:117] "RemoveContainer" containerID="4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268"
Jul 28 10:59:54.139743 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.139715    2578 scope.go:117] "RemoveContainer" containerID="4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268"
Jul 28 10:59:54.140038 ip-10-0-131-67 kubenswrapper[2578]: E0728 10:59:54.140018    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268\": container with ID starting with 4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268 not found: ID does not exist" containerID="4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268"
Jul 28 10:59:54.140106 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.140048    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268"} err="failed to get container status \"4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268\": rpc error: code = NotFound desc = could not find container \"4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268\": container with ID starting with 4fc8c17eb932a66c06d1f8d5f43264c6e2bfa11fbd3641e9d1f7a18f39ee7268 not found: ID does not exist"
Jul 28 10:59:54.151392 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.151361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"]
Jul 28 10:59:54.155683 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:54.155645    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4217be61-74d8bf6c6d-8hn78"]
Jul 28 10:59:55.493952 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:55.493917    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8009cd-1367-443c-9880-37cf6fc71c94" path="/var/lib/kubelet/pods/9e8009cd-1367-443c-9880-37cf6fc71c94/volumes"
Jul 28 10:59:58.046684 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:58.046641    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"]
Jul 28 10:59:58.048260 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:58.048231    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4217be61-5j27b"]
Jul 28 10:59:59.494749 ip-10-0-131-67 kubenswrapper[2578]: I0728 10:59:59.494709    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa" path="/var/lib/kubelet/pods/8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa/volumes"
Jul 28 11:00:00.134622 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.134581    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"]
Jul 28 11:00:00.134944 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.134930    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8009cd-1367-443c-9880-37cf6fc71c94" containerName="payload-processing"
Jul 28 11:00:00.135003 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.134946    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8009cd-1367-443c-9880-37cf6fc71c94" containerName="payload-processing"
Jul 28 11:00:00.135003 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.134963    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa" containerName="revoke-keys"
Jul 28 11:00:00.135003 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.134969    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa" containerName="revoke-keys"
Jul 28 11:00:00.135097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.135031    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f4b8bb2-7c2b-4bc6-ba09-b9922c1d6dfa" containerName="revoke-keys"
Jul 28 11:00:00.135097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.135043    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8009cd-1367-443c-9880-37cf6fc71c94" containerName="payload-processing"
Jul 28 11:00:00.139217 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.139189    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"]
Jul 28 11:00:00.139392 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.139373    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.142515 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.142476    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 11:00:00.142515 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.142503    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-b5796d54-openshift-default-dockercfg-mznn7\""
Jul 28 11:00:00.142743 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.142513    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"
Jul 28 11:00:00.144954 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.144934    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:00:00.154857 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.154771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"]
Jul 28 11:00:00.157681 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.157158    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"]
Jul 28 11:00:00.162345 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.162306    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"
Jul 28 11:00:00.302529 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-socket\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-credential-socket\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-istio-token\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302722 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrrv\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-kube-api-access-ccrrv\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302722 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-certs\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-data\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302946 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302867    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-envoy\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.302946 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda193e-5121-4f2f-851f-3c616718c210-istio-podinfo\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.303044 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.302968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda193e-5121-4f2f-851f-3c616718c210-istiod-ca-cert\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.303321 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.303299    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"]
Jul 28 11:00:00.305305 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:00:00.305259    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6058dd3_879d_4378_bfa1_da0cb87d35af.slice/crio-15e62bd282c749ac8e59ee05e00256dc5b97058006c51d7220b9bf39fe302851 WatchSource:0}: Error finding container 15e62bd282c749ac8e59ee05e00256dc5b97058006c51d7220b9bf39fe302851: Status 404 returned error can't find the container with id 15e62bd282c749ac8e59ee05e00256dc5b97058006c51d7220b9bf39fe302851
Jul 28 11:00:00.403821 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.403714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrrv\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-kube-api-access-ccrrv\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.403821 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.403780    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-certs\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.403821 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.403819    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-data\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.403885    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-envoy\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.403910    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda193e-5121-4f2f-851f-3c616718c210-istio-podinfo\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.403950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda193e-5121-4f2f-851f-3c616718c210-istiod-ca-cert\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404001    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-socket\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404032    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-credential-socket\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-istio-token\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404195    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-certs\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404227    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-data\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404936 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-credential-socket\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404936 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404489    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-socket\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.404936 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.404795    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda193e-5121-4f2f-851f-3c616718c210-istiod-ca-cert\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.406424 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.406397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-envoy\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.406526 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.406488    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda193e-5121-4f2f-851f-3c616718c210-istio-podinfo\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.412119 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.412091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-istio-token\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.412348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.412328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrrv\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-kube-api-access-ccrrv\") pod \"e2e-derive-b5796d54-openshift-default-874c89888-zsfd2\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") " pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.452913 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.452871    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:00.599131 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.599072    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"]
Jul 28 11:00:00.601351 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:00:00.601316    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dda193e_5121_4f2f_851f_3c616718c210.slice/crio-9c2a8a92d0065f67496dcfbdaaea1281360bd8e7c41320ab7166413066f07a1b WatchSource:0}: Error finding container 9c2a8a92d0065f67496dcfbdaaea1281360bd8e7c41320ab7166413066f07a1b: Status 404 returned error can't find the container with id 9c2a8a92d0065f67496dcfbdaaea1281360bd8e7c41320ab7166413066f07a1b
Jul 28 11:00:00.603580 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.603548    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:00:00.603673 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.603615    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:00:00.603673 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:00.603650    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:00:01.167243 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.167203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" event={"ID":"4dda193e-5121-4f2f-851f-3c616718c210","Type":"ContainerStarted","Data":"137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"}
Jul 28 11:00:01.167571 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.167261    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" event={"ID":"4dda193e-5121-4f2f-851f-3c616718c210","Type":"ContainerStarted","Data":"9c2a8a92d0065f67496dcfbdaaea1281360bd8e7c41320ab7166413066f07a1b"}
Jul 28 11:00:01.168954 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.168922    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7" event={"ID":"b6058dd3-879d-4378-bfa1-da0cb87d35af","Type":"ContainerStarted","Data":"824b808c0ed6416200cc0987d5d73e16be46543a0cc78e05a450e61ef0079b27"}
Jul 28 11:00:01.169081 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.168971    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7" event={"ID":"b6058dd3-879d-4378-bfa1-da0cb87d35af","Type":"ContainerStarted","Data":"15e62bd282c749ac8e59ee05e00256dc5b97058006c51d7220b9bf39fe302851"}
Jul 28 11:00:01.194523 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.194471    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podStartSLOduration=1.194453165 podStartE2EDuration="1.194453165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:00:01.193285262 +0000 UTC m=+2192.235377419" watchObservedRunningTime="2026-07-28 11:00:01.194453165 +0000 UTC m=+2192.236545317"
Jul 28 11:00:01.211709 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.211651    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7" podStartSLOduration=1.211630214 podStartE2EDuration="1.211630214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:00:01.210093731 +0000 UTC m=+2192.252185883" watchObservedRunningTime="2026-07-28 11:00:01.211630214 +0000 UTC m=+2192.253722397"
Jul 28 11:00:01.453945 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.453900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:01.459116 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:01.459088    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:02.175577 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:02.175536    2578 generic.go:356] "Generic (PLEG): container finished" podID="b6058dd3-879d-4378-bfa1-da0cb87d35af" containerID="824b808c0ed6416200cc0987d5d73e16be46543a0cc78e05a450e61ef0079b27" exitCode=0
Jul 28 11:00:02.175963 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:02.175622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7" event={"ID":"b6058dd3-879d-4378-bfa1-da0cb87d35af","Type":"ContainerDied","Data":"824b808c0ed6416200cc0987d5d73e16be46543a0cc78e05a450e61ef0079b27"}
Jul 28 11:00:02.175963 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:02.175831    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:02.176788 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:02.176770    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:03.319471 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.319442    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"
Jul 28 11:00:03.373868 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.373831    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"]
Jul 28 11:00:03.374176 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.374164    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6058dd3-879d-4378-bfa1-da0cb87d35af" containerName="cleanup"
Jul 28 11:00:03.374220 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.374178    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6058dd3-879d-4378-bfa1-da0cb87d35af" containerName="cleanup"
Jul 28 11:00:03.374255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.374238    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6058dd3-879d-4378-bfa1-da0cb87d35af" containerName="cleanup"
Jul 28 11:00:03.377084 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.377059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.380455 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.380421    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-b5796d54-dockercfg-q9fwg\""
Jul 28 11:00:03.380598 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.380578    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-b5796d54\""
Jul 28 11:00:03.385686 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.385657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"]
Jul 28 11:00:03.531889 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.531844    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mjmj\" (UniqueName: \"kubernetes.io/projected/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-kube-api-access-7mjmj\") pod \"payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") " pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.532051 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.531937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-plugins-config-volume\") pod \"payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") " pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.633245 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.633198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mjmj\" (UniqueName: \"kubernetes.io/projected/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-kube-api-access-7mjmj\") pod \"payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") " pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.633478 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.633333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-plugins-config-volume\") pod \"payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") " pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.634034 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.634012    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-plugins-config-volume\") pod \"payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") " pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.644766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.644738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mjmj\" (UniqueName: \"kubernetes.io/projected/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-kube-api-access-7mjmj\") pod \"payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") " pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.688757 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.688714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:03.832671 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:03.832636    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"]
Jul 28 11:00:03.836100 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:00:03.836076    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69b5ac65_8b08_45d2_ac07_64e9b7d312fb.slice/crio-3249f14ae99e2fe3d1f15e79455953e63d2aea8dce127d6cf92eda447cdeb633 WatchSource:0}: Error finding container 3249f14ae99e2fe3d1f15e79455953e63d2aea8dce127d6cf92eda447cdeb633: Status 404 returned error can't find the container with id 3249f14ae99e2fe3d1f15e79455953e63d2aea8dce127d6cf92eda447cdeb633
Jul 28 11:00:04.187164 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.187138    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7"
Jul 28 11:00:04.187373 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.187168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753940-z2bc7" event={"ID":"b6058dd3-879d-4378-bfa1-da0cb87d35af","Type":"ContainerDied","Data":"15e62bd282c749ac8e59ee05e00256dc5b97058006c51d7220b9bf39fe302851"}
Jul 28 11:00:04.187373 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.187217    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e62bd282c749ac8e59ee05e00256dc5b97058006c51d7220b9bf39fe302851"
Jul 28 11:00:04.188797 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.188767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc" event={"ID":"69b5ac65-8b08-45d2-ac07-64e9b7d312fb","Type":"ContainerStarted","Data":"b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10"}
Jul 28 11:00:04.188942 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.188818    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc" event={"ID":"69b5ac65-8b08-45d2-ac07-64e9b7d312fb","Type":"ContainerStarted","Data":"3249f14ae99e2fe3d1f15e79455953e63d2aea8dce127d6cf92eda447cdeb633"}
Jul 28 11:00:04.189078 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.189057    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:04.205997 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:04.205937    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc" podStartSLOduration=1.205920958 podStartE2EDuration="1.205920958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:00:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:00:04.204250565 +0000 UTC m=+2195.246342720" watchObservedRunningTime="2026-07-28 11:00:04.205920958 +0000 UTC m=+2195.248013111"
Jul 28 11:00:15.195704 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:15.195669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:17.177290 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.177237    2578 patch_prober.go:28] interesting pod/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 28 11:00:17.177718 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.177319    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 28 11:00:17.258470 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.258438    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dda193e-5121-4f2f-851f-3c616718c210" containerID="137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84" exitCode=0
Jul 28 11:00:17.258665 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.258518    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" event={"ID":"4dda193e-5121-4f2f-851f-3c616718c210","Type":"ContainerDied","Data":"137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"}
Jul 28 11:00:17.258960 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.258944    2578 scope.go:117] "RemoveContainer" containerID="137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"
Jul 28 11:00:17.260034 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.260011    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:00:17.260097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.260077    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:00:17.260133 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:17.260109    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:00:18.272019 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:18.271974    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" event={"ID":"4dda193e-5121-4f2f-851f-3c616718c210","Type":"ContainerStarted","Data":"0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048"}
Jul 28 11:00:18.453070 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:18.453033    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:18.453348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:18.453323    2578 patch_prober.go:28] interesting pod/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 28 11:00:18.453446 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:18.453387    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 28 11:00:19.453539 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:19.453499    2578 patch_prober.go:28] interesting pod/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 28 11:00:19.454042 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:19.453564    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 28 11:00:20.453624 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:20.453586    2578 patch_prober.go:28] interesting pod/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 28 11:00:20.454052 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:20.453656    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 28 11:00:21.453850 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:21.453806    2578 patch_prober.go:28] interesting pod/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body=
Jul 28 11:00:21.454329 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:21.453875    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused"
Jul 28 11:00:22.457870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:22.457837    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:22.458378 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:22.458123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:22.459068 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:22.459050    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:27.565606 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.565566    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"]
Jul 28 11:00:27.568981 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.568959    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:27.571976 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.571943    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 11:00:27.572443 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.572425    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:00:27.577576 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.577549    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"]
Jul 28 11:00:27.642664 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.642613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cf6468-2443-448d-816b-eb1082635c56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b5796d54-78gzt\" (UID: \"f7cf6468-2443-448d-816b-eb1082635c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:27.743481 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.743439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cf6468-2443-448d-816b-eb1082635c56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b5796d54-78gzt\" (UID: \"f7cf6468-2443-448d-816b-eb1082635c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:27.744068 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.744047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cf6468-2443-448d-816b-eb1082635c56-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b5796d54-78gzt\" (UID: \"f7cf6468-2443-448d-816b-eb1082635c56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:27.881547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:27.881454    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:28.023958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:28.023924    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"]
Jul 28 11:00:28.027041 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:00:28.027004    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7cf6468_2443_448d_816b_eb1082635c56.slice/crio-6647c58ff5d3084b15cea376ceaf565d1b78c38a35e7b6d98658dfdc46a75f63 WatchSource:0}: Error finding container 6647c58ff5d3084b15cea376ceaf565d1b78c38a35e7b6d98658dfdc46a75f63: Status 404 returned error can't find the container with id 6647c58ff5d3084b15cea376ceaf565d1b78c38a35e7b6d98658dfdc46a75f63
Jul 28 11:00:28.324515 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:28.324479    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt" event={"ID":"f7cf6468-2443-448d-816b-eb1082635c56","Type":"ContainerStarted","Data":"6647c58ff5d3084b15cea376ceaf565d1b78c38a35e7b6d98658dfdc46a75f63"}
Jul 28 11:00:29.331414 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:29.331351    2578 generic.go:356] "Generic (PLEG): container finished" podID="f7cf6468-2443-448d-816b-eb1082635c56" containerID="b85e292c7cc0274bfda2648cb12e81de249f31ea96df1485853470a5897d76b1" exitCode=0
Jul 28 11:00:29.331924 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:29.331511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt" event={"ID":"f7cf6468-2443-448d-816b-eb1082635c56","Type":"ContainerDied","Data":"b85e292c7cc0274bfda2648cb12e81de249f31ea96df1485853470a5897d76b1"}
Jul 28 11:00:30.472559 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:30.472530    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:30.570921 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:30.570883    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cf6468-2443-448d-816b-eb1082635c56-maas-api-service-ca\") pod \"f7cf6468-2443-448d-816b-eb1082635c56\" (UID: \"f7cf6468-2443-448d-816b-eb1082635c56\") "
Jul 28 11:00:30.571309 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:30.571255    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7cf6468-2443-448d-816b-eb1082635c56-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7cf6468-2443-448d-816b-eb1082635c56" (UID: "f7cf6468-2443-448d-816b-eb1082635c56"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:00:30.672161 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:30.672130    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7cf6468-2443-448d-816b-eb1082635c56-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:31.342649 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:31.342611    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"
Jul 28 11:00:31.342649 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:31.342628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt" event={"ID":"f7cf6468-2443-448d-816b-eb1082635c56","Type":"ContainerDied","Data":"6647c58ff5d3084b15cea376ceaf565d1b78c38a35e7b6d98658dfdc46a75f63"}
Jul 28 11:00:31.342867 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:31.342671    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6647c58ff5d3084b15cea376ceaf565d1b78c38a35e7b6d98658dfdc46a75f63"
Jul 28 11:00:38.015717 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.015681    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"]
Jul 28 11:00:38.016316 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.015902    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc" podUID="69b5ac65-8b08-45d2-ac07-64e9b7d312fb" containerName="payload-processing" containerID="cri-o://b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10" gracePeriod=30
Jul 28 11:00:38.269035 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.268960    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:38.343847 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.343808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mjmj\" (UniqueName: \"kubernetes.io/projected/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-kube-api-access-7mjmj\") pod \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") "
Jul 28 11:00:38.343847 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.343846    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-plugins-config-volume\") pod \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\" (UID: \"69b5ac65-8b08-45d2-ac07-64e9b7d312fb\") "
Jul 28 11:00:38.344315 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.344290    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "69b5ac65-8b08-45d2-ac07-64e9b7d312fb" (UID: "69b5ac65-8b08-45d2-ac07-64e9b7d312fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:00:38.346196 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.346174    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-kube-api-access-7mjmj" (OuterVolumeSpecName: "kube-api-access-7mjmj") pod "69b5ac65-8b08-45d2-ac07-64e9b7d312fb" (UID: "69b5ac65-8b08-45d2-ac07-64e9b7d312fb"). InnerVolumeSpecName "kube-api-access-7mjmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:00:38.376685 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.376646    2578 generic.go:356] "Generic (PLEG): container finished" podID="69b5ac65-8b08-45d2-ac07-64e9b7d312fb" containerID="b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10" exitCode=0
Jul 28 11:00:38.376879 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.376690    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"
Jul 28 11:00:38.376879 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.376722    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc" event={"ID":"69b5ac65-8b08-45d2-ac07-64e9b7d312fb","Type":"ContainerDied","Data":"b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10"}
Jul 28 11:00:38.376879 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.376774    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc" event={"ID":"69b5ac65-8b08-45d2-ac07-64e9b7d312fb","Type":"ContainerDied","Data":"3249f14ae99e2fe3d1f15e79455953e63d2aea8dce127d6cf92eda447cdeb633"}
Jul 28 11:00:38.376879 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.376795    2578 scope.go:117] "RemoveContainer" containerID="b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10"
Jul 28 11:00:38.385515 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.385494    2578 scope.go:117] "RemoveContainer" containerID="b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10"
Jul 28 11:00:38.385849 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:00:38.385832    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10\": container with ID starting with b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10 not found: ID does not exist" containerID="b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10"
Jul 28 11:00:38.385907 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.385858    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10"} err="failed to get container status \"b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10\": rpc error: code = NotFound desc = could not find container \"b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10\": container with ID starting with b1ca8aa900fe0f813807598df181b4141f3170b3207fa89a72f19f6f57e76e10 not found: ID does not exist"
Jul 28 11:00:38.397801 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.397765    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"]
Jul 28 11:00:38.401570 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.401540    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-b5796d54-5844f9b6cd-thlhc"]
Jul 28 11:00:38.445221 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.445185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mjmj\" (UniqueName: \"kubernetes.io/projected/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-kube-api-access-7mjmj\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:38.445221 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:38.445216    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69b5ac65-8b08-45d2-ac07-64e9b7d312fb-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:39.493696 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:39.493660    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69b5ac65-8b08-45d2-ac07-64e9b7d312fb" path="/var/lib/kubelet/pods/69b5ac65-8b08-45d2-ac07-64e9b7d312fb/volumes"
Jul 28 11:00:42.642196 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:42.642152    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"]
Jul 28 11:00:42.643946 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:42.643919    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b5796d54-78gzt"]
Jul 28 11:00:42.780346 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:42.780302    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"]
Jul 28 11:00:42.780631 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:42.780568    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy" containerID="cri-o://0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048" gracePeriod=30
Jul 28 11:00:43.494077 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:43.494038    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7cf6468-2443-448d-816b-eb1082635c56" path="/var/lib/kubelet/pods/f7cf6468-2443-448d-816b-eb1082635c56/volumes"
Jul 28 11:00:48.047039 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.046963    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:48.130857 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.130809    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-envoy\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.130888    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccrrv\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-kube-api-access-ccrrv\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.130919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-socket\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.130944    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-istio-token\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.130970    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda193e-5121-4f2f-851f-3c616718c210-istio-podinfo\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.130998    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-data\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.131060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda193e-5121-4f2f-851f-3c616718c210-istiod-ca-cert\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131433 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.131090    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-credential-socket\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131433 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.131143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-certs\") pod \"4dda193e-5121-4f2f-851f-3c616718c210\" (UID: \"4dda193e-5121-4f2f-851f-3c616718c210\") "
Jul 28 11:00:48.131433 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.131191    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:00:48.131815 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.131784    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:00:48.132041 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.132005    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dda193e-5121-4f2f-851f-3c616718c210-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:00:48.132313 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.132286    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:00:48.132457 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.132382    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.140455 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.132570    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-data" (OuterVolumeSpecName: "istio-data") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:00:48.140455 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.134521    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:00:48.140803 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.140630    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-istio-token" (OuterVolumeSpecName: "istio-token") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:00:48.141232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.141200    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dda193e-5121-4f2f-851f-3c616718c210-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:00:48.141232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.141215    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-kube-api-access-ccrrv" (OuterVolumeSpecName: "kube-api-access-ccrrv") pod "4dda193e-5121-4f2f-851f-3c616718c210" (UID: "4dda193e-5121-4f2f-851f-3c616718c210"). InnerVolumeSpecName "kube-api-access-ccrrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:00:48.233389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233347    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233383    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrrv\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-kube-api-access-ccrrv\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233393    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dda193e-5121-4f2f-851f-3c616718c210-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233634 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233401    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dda193e-5121-4f2f-851f-3c616718c210-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233634 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233410    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233634 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233417    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dda193e-5121-4f2f-851f-3c616718c210-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233634 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233424    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.233634 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.233432    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dda193e-5121-4f2f-851f-3c616718c210-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:00:48.424682 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.424644    2578 generic.go:356] "Generic (PLEG): container finished" podID="4dda193e-5121-4f2f-851f-3c616718c210" containerID="0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048" exitCode=0
Jul 28 11:00:48.424891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.424703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" event={"ID":"4dda193e-5121-4f2f-851f-3c616718c210","Type":"ContainerDied","Data":"0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048"}
Jul 28 11:00:48.424891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.424721    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"
Jul 28 11:00:48.424891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.424776    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2" event={"ID":"4dda193e-5121-4f2f-851f-3c616718c210","Type":"ContainerDied","Data":"9c2a8a92d0065f67496dcfbdaaea1281360bd8e7c41320ab7166413066f07a1b"}
Jul 28 11:00:48.424891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.424798    2578 scope.go:117] "RemoveContainer" containerID="0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048"
Jul 28 11:00:48.435933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.435906    2578 scope.go:117] "RemoveContainer" containerID="137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"
Jul 28 11:00:48.445160 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.445139    2578 scope.go:117] "RemoveContainer" containerID="0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048"
Jul 28 11:00:48.445522 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:00:48.445501    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048\": container with ID starting with 0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048 not found: ID does not exist" containerID="0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048"
Jul 28 11:00:48.445575 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.445534    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048"} err="failed to get container status \"0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048\": rpc error: code = NotFound desc = could not find container \"0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048\": container with ID starting with 0525bf901eaee5f64f1df641b70f6a375011d7d26347222e4111c5c7d434c048 not found: ID does not exist"
Jul 28 11:00:48.445575 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.445555    2578 scope.go:117] "RemoveContainer" containerID="137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"
Jul 28 11:00:48.445853 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:00:48.445830    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84\": container with ID starting with 137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84 not found: ID does not exist" containerID="137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"
Jul 28 11:00:48.445970 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.445856    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84"} err="failed to get container status \"137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84\": rpc error: code = NotFound desc = could not find container \"137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84\": container with ID starting with 137c0ad06eb89caaecbc85483e0ca4fd789f82ffa0360d759769f86c85632e84 not found: ID does not exist"
Jul 28 11:00:48.448604 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.448573    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"]
Jul 28 11:00:48.452225 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:48.452194    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-b5796d54-openshift-default-874c89888-zsfd2"]
Jul 28 11:00:49.494167 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:00:49.494086    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dda193e-5121-4f2f-851f-3c616718c210" path="/var/lib/kubelet/pods/4dda193e-5121-4f2f-851f-3c616718c210/volumes"
Jul 28 11:02:31.548937 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.548818    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"]
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549215    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69b5ac65-8b08-45d2-ac07-64e9b7d312fb" containerName="payload-processing"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549228    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="69b5ac65-8b08-45d2-ac07-64e9b7d312fb" containerName="payload-processing"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549250    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549256    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549264    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7cf6468-2443-448d-816b-eb1082635c56" containerName="revoke-keys"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549285    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7cf6468-2443-448d-816b-eb1082635c56" containerName="revoke-keys"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549344    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549354    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549362    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="69b5ac65-8b08-45d2-ac07-64e9b7d312fb" containerName="payload-processing"
Jul 28 11:02:31.549569 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.549368    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7cf6468-2443-448d-816b-eb1082635c56" containerName="revoke-keys"
Jul 28 11:02:31.552687 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.552665    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.555328 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.555299    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 11:02:31.555450 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.555434    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-df73bc23-openshift-default-dockercfg-m4mjp\""
Jul 28 11:02:31.578748 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.578715    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"]
Jul 28 11:02:31.642884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.642846    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-podinfo\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.642884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.642895    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-token\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643117 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.642918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjms\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-kube-api-access-nfjms\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643117 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.642945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-certs\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643117 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.643042    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d051bee2-24f6-487f-b07c-a7b18910fe1f-istiod-ca-cert\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643117 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.643092    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-data\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.643167    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-envoy\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.643221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-socket\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.643355 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.643251    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-credential-socket\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.743743 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743700    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-podinfo\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.743743 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-token\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743770    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjms\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-kube-api-access-nfjms\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743804    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-certs\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743838    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d051bee2-24f6-487f-b07c-a7b18910fe1f-istiod-ca-cert\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-data\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743918    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-envoy\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-socket\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.743975    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-credential-socket\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744453 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.744342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-certs\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744561 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.744538    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-data\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744625 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.744570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-credential-socket\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.744625 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.744591    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-socket\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.745027 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.745000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d051bee2-24f6-487f-b07c-a7b18910fe1f-istiod-ca-cert\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.746479 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.746457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-envoy\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.746606 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.746516    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-podinfo\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.753661 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.753636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-token\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.754081 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.754062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjms\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-kube-api-access-nfjms\") pod \"e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") " pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:31.863552 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:31.863456    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:32.011674 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.011641    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"]
Jul 28 11:02:32.014095 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:02:32.014066    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd051bee2_24f6_487f_b07c_a7b18910fe1f.slice/crio-132ade2554da7e502269219b7fc13937cfb777ee1822ef601884539c5ec1d0d2 WatchSource:0}: Error finding container 132ade2554da7e502269219b7fc13937cfb777ee1822ef601884539c5ec1d0d2: Status 404 returned error can't find the container with id 132ade2554da7e502269219b7fc13937cfb777ee1822ef601884539c5ec1d0d2
Jul 28 11:02:32.016592 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.016552    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:02:32.016695 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.016635    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:02:32.016695 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.016674    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:02:32.917136 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.917096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" event={"ID":"d051bee2-24f6-487f-b07c-a7b18910fe1f","Type":"ContainerStarted","Data":"364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"}
Jul 28 11:02:32.917651 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.917154    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" event={"ID":"d051bee2-24f6-487f-b07c-a7b18910fe1f","Type":"ContainerStarted","Data":"132ade2554da7e502269219b7fc13937cfb777ee1822ef601884539c5ec1d0d2"}
Jul 28 11:02:32.938565 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:32.938502    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podStartSLOduration=1.938482467 podStartE2EDuration="1.938482467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:02:32.936447724 +0000 UTC m=+2343.978539889" watchObservedRunningTime="2026-07-28 11:02:32.938482467 +0000 UTC m=+2343.980574619"
Jul 28 11:02:33.864089 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:33.864046    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:33.869120 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:33.869095    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:33.921749 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:33.921717    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:33.922672 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:33.922654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:35.774324 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.774289    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"]
Jul 28 11:02:35.774713 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.774645    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy"
Jul 28 11:02:35.774713 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.774656    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dda193e-5121-4f2f-851f-3c616718c210" containerName="istio-proxy"
Jul 28 11:02:35.777738 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.777719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:35.780654 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.780630    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-df73bc23\""
Jul 28 11:02:35.780794 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.780632    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-df73bc23-dockercfg-bxcsj\""
Jul 28 11:02:35.790010 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.789984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"]
Jul 28 11:02:35.877856 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.877816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7w65\" (UniqueName: \"kubernetes.io/projected/264570f4-a7e5-468a-8f8d-91ae71bd26ac-kube-api-access-g7w65\") pod \"payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") " pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:35.878042 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.877893    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264570f4-a7e5-468a-8f8d-91ae71bd26ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") " pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:35.979068 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.979031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7w65\" (UniqueName: \"kubernetes.io/projected/264570f4-a7e5-468a-8f8d-91ae71bd26ac-kube-api-access-g7w65\") pod \"payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") " pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:35.979246 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.979127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264570f4-a7e5-468a-8f8d-91ae71bd26ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") " pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:35.979862 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.979840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264570f4-a7e5-468a-8f8d-91ae71bd26ac-plugins-config-volume\") pod \"payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") " pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:35.987123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:35.987100    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7w65\" (UniqueName: \"kubernetes.io/projected/264570f4-a7e5-468a-8f8d-91ae71bd26ac-kube-api-access-g7w65\") pod \"payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") " pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:36.088100 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:36.088005    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:36.230477 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:36.230389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"]
Jul 28 11:02:36.232961 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:02:36.232935    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264570f4_a7e5_468a_8f8d_91ae71bd26ac.slice/crio-b2d575cc8137dbaeecb2280ac1a10a0c8286ac01383a1b4b858f4e6ff3a381af WatchSource:0}: Error finding container b2d575cc8137dbaeecb2280ac1a10a0c8286ac01383a1b4b858f4e6ff3a381af: Status 404 returned error can't find the container with id b2d575cc8137dbaeecb2280ac1a10a0c8286ac01383a1b4b858f4e6ff3a381af
Jul 28 11:02:36.938891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:36.938855    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85" event={"ID":"264570f4-a7e5-468a-8f8d-91ae71bd26ac","Type":"ContainerStarted","Data":"eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a"}
Jul 28 11:02:36.939408 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:36.938909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85" event={"ID":"264570f4-a7e5-468a-8f8d-91ae71bd26ac","Type":"ContainerStarted","Data":"b2d575cc8137dbaeecb2280ac1a10a0c8286ac01383a1b4b858f4e6ff3a381af"}
Jul 28 11:02:36.939408 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:36.938981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:36.957977 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:36.957910    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85" podStartSLOduration=1.957887836 podStartE2EDuration="1.957887836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:02:36.957037798 +0000 UTC m=+2347.999129952" watchObservedRunningTime="2026-07-28 11:02:36.957887836 +0000 UTC m=+2347.999979991"
Jul 28 11:02:46.793231 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.793189    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"]
Jul 28 11:02:46.799948 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.799904    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:46.803152 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.803120    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:02:46.803376 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.803350    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 11:02:46.803816 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.803785    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"]
Jul 28 11:02:46.878725 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.878691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e2705b-62b3-4be0-8129-652eb3e80175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7\" (UID: \"c5e2705b-62b3-4be0-8129-652eb3e80175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:46.979801 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.979762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e2705b-62b3-4be0-8129-652eb3e80175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7\" (UID: \"c5e2705b-62b3-4be0-8129-652eb3e80175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:46.980416 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:46.980396    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e2705b-62b3-4be0-8129-652eb3e80175-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7\" (UID: \"c5e2705b-62b3-4be0-8129-652eb3e80175\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:47.124002 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:47.123865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:47.260434 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:47.260405    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"]
Jul 28 11:02:47.262891 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:02:47.262853    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e2705b_62b3_4be0_8129_652eb3e80175.slice/crio-1ecc3d772feed5388ff708e95e860ab01e3e442173bb13097c009930898b0799 WatchSource:0}: Error finding container 1ecc3d772feed5388ff708e95e860ab01e3e442173bb13097c009930898b0799: Status 404 returned error can't find the container with id 1ecc3d772feed5388ff708e95e860ab01e3e442173bb13097c009930898b0799
Jul 28 11:02:47.945373 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:47.945345    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:47.993593 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:47.993556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7" event={"ID":"c5e2705b-62b3-4be0-8129-652eb3e80175","Type":"ContainerStarted","Data":"ed3642c91ce2537a5dcd12001810cf2e2fa8272b783d469c5769a048e2864f32"}
Jul 28 11:02:47.993773 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:47.993616    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7" event={"ID":"c5e2705b-62b3-4be0-8129-652eb3e80175","Type":"ContainerStarted","Data":"1ecc3d772feed5388ff708e95e860ab01e3e442173bb13097c009930898b0799"}
Jul 28 11:02:48.009675 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:48.009616    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7" podStartSLOduration=2.00959769 podStartE2EDuration="2.00959769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:02:48.008086542 +0000 UTC m=+2359.050178733" watchObservedRunningTime="2026-07-28 11:02:48.00959769 +0000 UTC m=+2359.051689842"
Jul 28 11:02:49.000802 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:49.000768    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5e2705b-62b3-4be0-8129-652eb3e80175" containerID="ed3642c91ce2537a5dcd12001810cf2e2fa8272b783d469c5769a048e2864f32" exitCode=0
Jul 28 11:02:49.001187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:49.000822    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7" event={"ID":"c5e2705b-62b3-4be0-8129-652eb3e80175","Type":"ContainerDied","Data":"ed3642c91ce2537a5dcd12001810cf2e2fa8272b783d469c5769a048e2864f32"}
Jul 28 11:02:49.749250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:49.748229    2578 patch_prober.go:28] interesting pod/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:39668->10.134.0.47:15021: read: connection reset by peer" start-of-body=
Jul 28 11:02:49.749250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:49.748317    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": read tcp 10.134.0.2:39668->10.134.0.47:15021: read: connection reset by peer"
Jul 28 11:02:50.006932 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.006837    2578 generic.go:356] "Generic (PLEG): container finished" podID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerID="364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef" exitCode=0
Jul 28 11:02:50.006932 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.006914    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" event={"ID":"d051bee2-24f6-487f-b07c-a7b18910fe1f","Type":"ContainerDied","Data":"364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"}
Jul 28 11:02:50.007924 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.007539    2578 scope.go:117] "RemoveContainer" containerID="364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"
Jul 28 11:02:50.008860 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.008831    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:02:50.009001 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.008897    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:02:50.009001 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.008935    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:02:50.162795 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.162770    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:50.304607 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.304509    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e2705b-62b3-4be0-8129-652eb3e80175-maas-api-service-ca\") pod \"c5e2705b-62b3-4be0-8129-652eb3e80175\" (UID: \"c5e2705b-62b3-4be0-8129-652eb3e80175\") "
Jul 28 11:02:50.304898 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.304871    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e2705b-62b3-4be0-8129-652eb3e80175-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5e2705b-62b3-4be0-8129-652eb3e80175" (UID: "c5e2705b-62b3-4be0-8129-652eb3e80175"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:02:50.405955 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:50.405916    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5e2705b-62b3-4be0-8129-652eb3e80175-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:02:51.018472 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.018425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" event={"ID":"d051bee2-24f6-487f-b07c-a7b18910fe1f","Type":"ContainerStarted","Data":"ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619"}
Jul 28 11:02:51.021148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.021106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7" event={"ID":"c5e2705b-62b3-4be0-8129-652eb3e80175","Type":"ContainerDied","Data":"1ecc3d772feed5388ff708e95e860ab01e3e442173bb13097c009930898b0799"}
Jul 28 11:02:51.021319 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.021172    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ecc3d772feed5388ff708e95e860ab01e3e442173bb13097c009930898b0799"
Jul 28 11:02:51.021319 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.021232    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"
Jul 28 11:02:51.864474 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.864428    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:51.864754 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.864730    2578 patch_prober.go:28] interesting pod/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 28 11:02:51.864839 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:51.864788    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 28 11:02:52.863997 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:52.863957    2578 patch_prober.go:28] interesting pod/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 28 11:02:52.864417 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:52.864017    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 28 11:02:53.864826 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:53.864783    2578 patch_prober.go:28] interesting pod/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 28 11:02:53.865351 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:53.864852    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 28 11:02:54.864593 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:54.864549    2578 patch_prober.go:28] interesting pod/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body=
Jul 28 11:02:54.864786 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:54.864622    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused"
Jul 28 11:02:55.867621 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:55.867589    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:55.868021 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:55.867961    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:55.868751 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:55.868733    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:02:57.227427 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.227391    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"]
Jul 28 11:02:57.227910 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.227641    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85" podUID="264570f4-a7e5-468a-8f8d-91ae71bd26ac" containerName="payload-processing" containerID="cri-o://eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a" gracePeriod=30
Jul 28 11:02:57.497123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.497095    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:57.568156 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.568120    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264570f4-a7e5-468a-8f8d-91ae71bd26ac-plugins-config-volume\") pod \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") "
Jul 28 11:02:57.568354 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.568193    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7w65\" (UniqueName: \"kubernetes.io/projected/264570f4-a7e5-468a-8f8d-91ae71bd26ac-kube-api-access-g7w65\") pod \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\" (UID: \"264570f4-a7e5-468a-8f8d-91ae71bd26ac\") "
Jul 28 11:02:57.568551 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.568527    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264570f4-a7e5-468a-8f8d-91ae71bd26ac-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "264570f4-a7e5-468a-8f8d-91ae71bd26ac" (UID: "264570f4-a7e5-468a-8f8d-91ae71bd26ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:02:57.570438 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.570409    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/264570f4-a7e5-468a-8f8d-91ae71bd26ac-kube-api-access-g7w65" (OuterVolumeSpecName: "kube-api-access-g7w65") pod "264570f4-a7e5-468a-8f8d-91ae71bd26ac" (UID: "264570f4-a7e5-468a-8f8d-91ae71bd26ac"). InnerVolumeSpecName "kube-api-access-g7w65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:02:57.669457 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.669413    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7w65\" (UniqueName: \"kubernetes.io/projected/264570f4-a7e5-468a-8f8d-91ae71bd26ac-kube-api-access-g7w65\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:02:57.669457 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:57.669448    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/264570f4-a7e5-468a-8f8d-91ae71bd26ac-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:02:58.061151 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.061115    2578 generic.go:356] "Generic (PLEG): container finished" podID="264570f4-a7e5-468a-8f8d-91ae71bd26ac" containerID="eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a" exitCode=0
Jul 28 11:02:58.061151 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.061155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"
Jul 28 11:02:58.061389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.061198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85" event={"ID":"264570f4-a7e5-468a-8f8d-91ae71bd26ac","Type":"ContainerDied","Data":"eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a"}
Jul 28 11:02:58.061389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.061248    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85" event={"ID":"264570f4-a7e5-468a-8f8d-91ae71bd26ac","Type":"ContainerDied","Data":"b2d575cc8137dbaeecb2280ac1a10a0c8286ac01383a1b4b858f4e6ff3a381af"}
Jul 28 11:02:58.061389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.061288    2578 scope.go:117] "RemoveContainer" containerID="eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a"
Jul 28 11:02:58.070677 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.070658    2578 scope.go:117] "RemoveContainer" containerID="eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a"
Jul 28 11:02:58.070967 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:02:58.070943    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a\": container with ID starting with eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a not found: ID does not exist" containerID="eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a"
Jul 28 11:02:58.071015 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.070976    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a"} err="failed to get container status \"eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a\": rpc error: code = NotFound desc = could not find container \"eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a\": container with ID starting with eaa64ef0a8c39e27886c2fbb4a1f8df383813fee005389c2b8a05c7e8f78e86a not found: ID does not exist"
Jul 28 11:02:58.083041 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.083005    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"]
Jul 28 11:02:58.084545 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:58.084524    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-df73bc23-7f65fd4b66-h5h85"]
Jul 28 11:02:59.494132 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:02:59.494098    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264570f4-a7e5-468a-8f8d-91ae71bd26ac" path="/var/lib/kubelet/pods/264570f4-a7e5-468a-8f8d-91ae71bd26ac/volumes"
Jul 28 11:03:01.875795 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:01.875757    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"]
Jul 28 11:03:01.878824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:01.878788    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-df73bc23-wp8j7"]
Jul 28 11:03:03.494148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:03.494113    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e2705b-62b3-4be0-8129-652eb3e80175" path="/var/lib/kubelet/pods/c5e2705b-62b3-4be0-8129-652eb3e80175/volumes"
Jul 28 11:03:08.544296 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:08.543328    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"]
Jul 28 11:03:08.544296 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:08.543644    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" containerID="cri-o://ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619" gracePeriod=30
Jul 28 11:03:10.870357 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:10.870322    2578 patch_prober.go:28] interesting pod/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 11:03:10.870750 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:10.870381    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 11:03:13.804702 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.804677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:03:13.914722 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-certs\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914722 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfjms\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-kube-api-access-nfjms\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914722 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914706    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-credential-socket\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914733    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d051bee2-24f6-487f-b07c-a7b18910fe1f-istiod-ca-cert\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914767    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-envoy\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914800    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-data\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914835    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-token\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914894    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-podinfo\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:03:13.914987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.914973    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-socket\") pod \"d051bee2-24f6-487f-b07c-a7b18910fe1f\" (UID: \"d051bee2-24f6-487f-b07c-a7b18910fe1f\") "
Jul 28 11:03:13.915365 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.915230    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:13.915365 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.915246    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d051bee2-24f6-487f-b07c-a7b18910fe1f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:03:13.915542 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.915504    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:03:13.915542 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.915518    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:03:13.915980 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.915950    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-data" (OuterVolumeSpecName: "istio-data") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:03:13.917072 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.917044    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-kube-api-access-nfjms" (OuterVolumeSpecName: "kube-api-access-nfjms") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "kube-api-access-nfjms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:03:13.917360 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.917340    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:03:13.917628 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.917604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:03:13.917694 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:13.917626    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-token" (OuterVolumeSpecName: "istio-token") pod "d051bee2-24f6-487f-b07c-a7b18910fe1f" (UID: "d051bee2-24f6-487f-b07c-a7b18910fe1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:03:14.016261 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016213    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016261 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016251    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016261 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016260    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016298    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfjms\" (UniqueName: \"kubernetes.io/projected/d051bee2-24f6-487f-b07c-a7b18910fe1f-kube-api-access-nfjms\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016314    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016328    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d051bee2-24f6-487f-b07c-a7b18910fe1f-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016340    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.016547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.016350    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d051bee2-24f6-487f-b07c-a7b18910fe1f-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:03:14.138859 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.138822    2578 generic.go:356] "Generic (PLEG): container finished" podID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerID="ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619" exitCode=0
Jul 28 11:03:14.139058 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.138868    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"
Jul 28 11:03:14.139058 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.138920    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" event={"ID":"d051bee2-24f6-487f-b07c-a7b18910fe1f","Type":"ContainerDied","Data":"ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619"}
Jul 28 11:03:14.139058 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.138985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk" event={"ID":"d051bee2-24f6-487f-b07c-a7b18910fe1f","Type":"ContainerDied","Data":"132ade2554da7e502269219b7fc13937cfb777ee1822ef601884539c5ec1d0d2"}
Jul 28 11:03:14.139058 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.139020    2578 scope.go:117] "RemoveContainer" containerID="ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619"
Jul 28 11:03:14.148724 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.148703    2578 scope.go:117] "RemoveContainer" containerID="364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"
Jul 28 11:03:14.157760 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.157737    2578 scope.go:117] "RemoveContainer" containerID="ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619"
Jul 28 11:03:14.158060 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:03:14.158043    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619\": container with ID starting with ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619 not found: ID does not exist" containerID="ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619"
Jul 28 11:03:14.158106 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.158071    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619"} err="failed to get container status \"ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619\": rpc error: code = NotFound desc = could not find container \"ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619\": container with ID starting with ccb6077e7e23d268c5f8760d019a7933b627209c607b1bad4ee4f5dda9503619 not found: ID does not exist"
Jul 28 11:03:14.158106 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.158089    2578 scope.go:117] "RemoveContainer" containerID="364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"
Jul 28 11:03:14.158365 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:03:14.158345    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef\": container with ID starting with 364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef not found: ID does not exist" containerID="364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"
Jul 28 11:03:14.158430 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.158371    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef"} err="failed to get container status \"364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef\": rpc error: code = NotFound desc = could not find container \"364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef\": container with ID starting with 364b25dc69fc6e42cee78c9eeaaf37a0bf827177d46caa854c65a6bbd39c6bef not found: ID does not exist"
Jul 28 11:03:14.161685 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.161652    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"]
Jul 28 11:03:14.165685 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:14.165616    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-df73bc23-openshift-default-86c6bd9ffb-ws5pk"]
Jul 28 11:03:15.493964 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:15.493929    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" path="/var/lib/kubelet/pods/d051bee2-24f6-487f-b07c-a7b18910fe1f/volumes"
Jul 28 11:03:29.084651 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.084598    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"]
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.084974    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e2705b-62b3-4be0-8129-652eb3e80175" containerName="revoke-keys"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.084987    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e2705b-62b3-4be0-8129-652eb3e80175" containerName="revoke-keys"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.084995    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085000    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085016    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085021    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085034    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264570f4-a7e5-468a-8f8d-91ae71bd26ac" containerName="payload-processing"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085039    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="264570f4-a7e5-468a-8f8d-91ae71bd26ac" containerName="payload-processing"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085105    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e2705b-62b3-4be0-8129-652eb3e80175" containerName="revoke-keys"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085117    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085128    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="264570f4-a7e5-468a-8f8d-91ae71bd26ac" containerName="payload-processing"
Jul 28 11:03:29.085148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.085139    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d051bee2-24f6-487f-b07c-a7b18910fe1f" containerName="istio-proxy"
Jul 28 11:03:29.088631 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.088609    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.091404 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.091382    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 11:03:29.091537 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.091426    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0b092d88-openshift-default-dockercfg-bcr7f\""
Jul 28 11:03:29.100340 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.100308    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"]
Jul 28 11:03:29.154865 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.154825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-token\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.154865 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.154873    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thn5x\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-kube-api-access-thn5x\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.154899    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-data\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.154992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-envoy\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.155024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-certs\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.155069    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istiod-ca-cert\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.155101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-socket\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.155123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-credential-socket\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.155400 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.155140    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-podinfo\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.255962 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.255916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istiod-ca-cert\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.255962 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.255968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-socket\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.255989    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-credential-socket\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256006    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-podinfo\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-token\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thn5x\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-kube-api-access-thn5x\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256204    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-data\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256538 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-envoy\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256538 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-certs\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256538 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256462    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-credential-socket\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256706 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-socket\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256706 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256636    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-data\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256737    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-certs\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.256784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.256778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istiod-ca-cert\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.258625 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.258606    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-envoy\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.258766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.258749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-podinfo\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.264596 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.264573    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-token\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.264718 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.264664    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thn5x\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-kube-api-access-thn5x\") pod \"e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") " pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.402341 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.402226    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0b092d88-openshift-default-dockercfg-bcr7f\""
Jul 28 11:03:29.409520 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.409492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:29.561090 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.561052    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"]
Jul 28 11:03:29.563225 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:03:29.563201    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cdaf46_8bbb_41b1_b362_c9d6f75689f9.slice/crio-febfa00975c8d68438899f045907e9d8a1aa5a7e1b90e6ada070a2333a204f4d WatchSource:0}: Error finding container febfa00975c8d68438899f045907e9d8a1aa5a7e1b90e6ada070a2333a204f4d: Status 404 returned error can't find the container with id febfa00975c8d68438899f045907e9d8a1aa5a7e1b90e6ada070a2333a204f4d
Jul 28 11:03:29.565701 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.565669    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:03:29.565782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.565734    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:03:29.565782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.565762    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:03:29.975166 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.975127    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:03:29.983975 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:29.983945    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:03:30.225700 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:30.225601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" event={"ID":"96cdaf46-8bbb-41b1-b362-c9d6f75689f9","Type":"ContainerStarted","Data":"56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"}
Jul 28 11:03:30.225700 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:30.225660    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" event={"ID":"96cdaf46-8bbb-41b1-b362-c9d6f75689f9","Type":"ContainerStarted","Data":"febfa00975c8d68438899f045907e9d8a1aa5a7e1b90e6ada070a2333a204f4d"}
Jul 28 11:03:30.246949 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:30.246890    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podStartSLOduration=1.246875441 podStartE2EDuration="1.246875441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:03:30.246056945 +0000 UTC m=+2401.288149118" watchObservedRunningTime="2026-07-28 11:03:30.246875441 +0000 UTC m=+2401.288967594"
Jul 28 11:03:30.410499 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:30.410456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:30.415256 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:30.415226    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:31.231301 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:31.231253    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:31.232197 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:31.232179    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:33.505751 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.505713    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"]
Jul 28 11:03:33.517085 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.517045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.521951 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.521904    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0b092d88\""
Jul 28 11:03:33.522131 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.522079    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0b092d88-dockercfg-fb94k\""
Jul 28 11:03:33.531492 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.531452    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"]
Jul 28 11:03:33.596645 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.596603    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qnhm\" (UniqueName: \"kubernetes.io/projected/d38c3c1e-9975-48d8-b81f-2a8b4c333911-kube-api-access-2qnhm\") pod \"payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") " pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.596820 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.596671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38c3c1e-9975-48d8-b81f-2a8b4c333911-plugins-config-volume\") pod \"payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") " pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.697535 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.697496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qnhm\" (UniqueName: \"kubernetes.io/projected/d38c3c1e-9975-48d8-b81f-2a8b4c333911-kube-api-access-2qnhm\") pod \"payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") " pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.697766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.697559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38c3c1e-9975-48d8-b81f-2a8b4c333911-plugins-config-volume\") pod \"payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") " pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.698303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.698258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38c3c1e-9975-48d8-b81f-2a8b4c333911-plugins-config-volume\") pod \"payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") " pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.711187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.711160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qnhm\" (UniqueName: \"kubernetes.io/projected/d38c3c1e-9975-48d8-b81f-2a8b4c333911-kube-api-access-2qnhm\") pod \"payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") " pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.828176 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.828071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:33.981650 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:33.981618    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"]
Jul 28 11:03:33.984225 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:03:33.984195    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd38c3c1e_9975_48d8_b81f_2a8b4c333911.slice/crio-acc4aed3346c291947474ea7c83b79594e0348397b34ca814a8024de9c242138 WatchSource:0}: Error finding container acc4aed3346c291947474ea7c83b79594e0348397b34ca814a8024de9c242138: Status 404 returned error can't find the container with id acc4aed3346c291947474ea7c83b79594e0348397b34ca814a8024de9c242138
Jul 28 11:03:34.247258 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:34.247219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4" event={"ID":"d38c3c1e-9975-48d8-b81f-2a8b4c333911","Type":"ContainerStarted","Data":"6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7"}
Jul 28 11:03:34.247496 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:34.247296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4" event={"ID":"d38c3c1e-9975-48d8-b81f-2a8b4c333911","Type":"ContainerStarted","Data":"acc4aed3346c291947474ea7c83b79594e0348397b34ca814a8024de9c242138"}
Jul 28 11:03:34.247496 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:34.247323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:34.271288 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:34.271229    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4" podStartSLOduration=1.271213379 podStartE2EDuration="1.271213379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:03:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:03:34.270698158 +0000 UTC m=+2405.312790315" watchObservedRunningTime="2026-07-28 11:03:34.271213379 +0000 UTC m=+2405.313305532"
Jul 28 11:03:42.138450 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.138413    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"]
Jul 28 11:03:42.141948 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.141925    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.145543 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.145511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0b092d88\"/\"kube-root-ca.crt\""
Jul 28 11:03:42.145681 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.145516    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0b092d88\"/\"e2e-lifecycle-model-0b092d88-kserve-self-signed-certs\""
Jul 28 11:03:42.145681 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.145611    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-0b092d88\"/\"openshift-service-ca.crt\""
Jul 28 11:03:42.145797 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.145779    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-0b092d88\"/\"default-dockercfg-9q45v\""
Jul 28 11:03:42.151561 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.151530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"]
Jul 28 11:03:42.274179 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.274144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a3a548-cd7c-4c29-abcc-78230ff10359-tls-certs\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.274397 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.274221    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-dshm\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.274397 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.274281    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-tmp-dir\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.274397 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.274344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-home\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.274397 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.274372    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-model-cache\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.274599 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.274424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfsk2\" (UniqueName: \"kubernetes.io/projected/41a3a548-cd7c-4c29-abcc-78230ff10359-kube-api-access-sfsk2\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375470 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375433    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-tmp-dir\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375470 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375484    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-home\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-model-cache\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfsk2\" (UniqueName: \"kubernetes.io/projected/41a3a548-cd7c-4c29-abcc-78230ff10359-kube-api-access-sfsk2\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375578    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a3a548-cd7c-4c29-abcc-78230ff10359-tls-certs\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-dshm\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375969 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375916    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-tmp-dir\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.375969 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375943    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-model-cache\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.376071 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.375972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-home\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.378125 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.378103    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-dshm\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.378439 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.378420    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a3a548-cd7c-4c29-abcc-78230ff10359-tls-certs\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.385215 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.385187    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfsk2\" (UniqueName: \"kubernetes.io/projected/41a3a548-cd7c-4c29-abcc-78230ff10359-kube-api-access-sfsk2\") pod \"e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") " pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.452573 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.452530    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:42.625297 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:42.621492    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"]
Jul 28 11:03:42.632441 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:03:42.632407    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41a3a548_cd7c_4c29_abcc_78230ff10359.slice/crio-97db6c87b91e5c5e620d2abb3ab9892454e2ffa596c2f12e8b2c4f42e76293d5 WatchSource:0}: Error finding container 97db6c87b91e5c5e620d2abb3ab9892454e2ffa596c2f12e8b2c4f42e76293d5: Status 404 returned error can't find the container with id 97db6c87b91e5c5e620d2abb3ab9892454e2ffa596c2f12e8b2c4f42e76293d5
Jul 28 11:03:43.291980 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:43.291935    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8" event={"ID":"41a3a548-cd7c-4c29-abcc-78230ff10359","Type":"ContainerStarted","Data":"421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05"}
Jul 28 11:03:43.292396 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:43.291984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8" event={"ID":"41a3a548-cd7c-4c29-abcc-78230ff10359","Type":"ContainerStarted","Data":"97db6c87b91e5c5e620d2abb3ab9892454e2ffa596c2f12e8b2c4f42e76293d5"}
Jul 28 11:03:43.311728 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:43.311657    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8" podStartSLOduration=1.311633989 podStartE2EDuration="1.311633989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:03:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:03:43.309291645 +0000 UTC m=+2414.351383800" watchObservedRunningTime="2026-07-28 11:03:43.311633989 +0000 UTC m=+2414.353726145"
Jul 28 11:03:45.254123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:45.254084    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:03:47.273467 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:47.273424    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 11:03:47.273862 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:47.273483    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 11:03:48.319936 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:48.319892    2578 generic.go:356] "Generic (PLEG): container finished" podID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerID="56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f" exitCode=0
Jul 28 11:03:48.320409 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:48.319940    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" event={"ID":"96cdaf46-8bbb-41b1-b362-c9d6f75689f9","Type":"ContainerDied","Data":"56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"}
Jul 28 11:03:48.320409 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:48.320344    2578 scope.go:117] "RemoveContainer" containerID="56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"
Jul 28 11:03:48.321369 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:48.321338    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:03:48.321490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:48.321427    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:03:48.321490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:48.321474    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:03:49.331001 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:49.330909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" event={"ID":"96cdaf46-8bbb-41b1-b362-c9d6f75689f9","Type":"ContainerStarted","Data":"7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93"}
Jul 28 11:03:49.410623 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:49.410583    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:49.411160 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:49.411132    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 28 11:03:49.411475 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:49.411443    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 28 11:03:50.410609 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:50.410552    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 28 11:03:50.411150 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:50.410648    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 28 11:03:51.410772 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:51.410725    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 28 11:03:51.411330 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:51.410795    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 28 11:03:52.410613 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:52.410555    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 28 11:03:52.410822 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:52.410647    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 28 11:03:52.453380 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:52.453331    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:52.453380 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:52.453389    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:52.466201 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:52.466160    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:53.360045 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:53.360019    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:03:53.410848 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:53.410801    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Jul 28 11:03:53.411357 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:53.410869    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Jul 28 11:03:54.413658 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:54.413617    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:54.414123 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:54.413900    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:03:54.414736 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:03:54.414719    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:04:26.924374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:26.924340    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"]
Jul 28 11:04:26.924959 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:26.924551    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4" podUID="d38c3c1e-9975-48d8-b81f-2a8b4c333911" containerName="payload-processing" containerID="cri-o://6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7" gracePeriod=30
Jul 28 11:04:27.179382 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.179303    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:04:27.288778 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.288736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qnhm\" (UniqueName: \"kubernetes.io/projected/d38c3c1e-9975-48d8-b81f-2a8b4c333911-kube-api-access-2qnhm\") pod \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") "
Jul 28 11:04:27.288968 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.288802    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38c3c1e-9975-48d8-b81f-2a8b4c333911-plugins-config-volume\") pod \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\" (UID: \"d38c3c1e-9975-48d8-b81f-2a8b4c333911\") "
Jul 28 11:04:27.289193 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.289165    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d38c3c1e-9975-48d8-b81f-2a8b4c333911-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d38c3c1e-9975-48d8-b81f-2a8b4c333911" (UID: "d38c3c1e-9975-48d8-b81f-2a8b4c333911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:04:27.291089 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.291062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d38c3c1e-9975-48d8-b81f-2a8b4c333911-kube-api-access-2qnhm" (OuterVolumeSpecName: "kube-api-access-2qnhm") pod "d38c3c1e-9975-48d8-b81f-2a8b4c333911" (UID: "d38c3c1e-9975-48d8-b81f-2a8b4c333911"). InnerVolumeSpecName "kube-api-access-2qnhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:04:27.389818 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.389777    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qnhm\" (UniqueName: \"kubernetes.io/projected/d38c3c1e-9975-48d8-b81f-2a8b4c333911-kube-api-access-2qnhm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:27.389818 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.389809    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d38c3c1e-9975-48d8-b81f-2a8b4c333911-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:27.522476 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.522442    2578 generic.go:356] "Generic (PLEG): container finished" podID="d38c3c1e-9975-48d8-b81f-2a8b4c333911" containerID="6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7" exitCode=0
Jul 28 11:04:27.522664 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.522489    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"
Jul 28 11:04:27.522664 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.522523    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4" event={"ID":"d38c3c1e-9975-48d8-b81f-2a8b4c333911","Type":"ContainerDied","Data":"6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7"}
Jul 28 11:04:27.522664 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.522586    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4" event={"ID":"d38c3c1e-9975-48d8-b81f-2a8b4c333911","Type":"ContainerDied","Data":"acc4aed3346c291947474ea7c83b79594e0348397b34ca814a8024de9c242138"}
Jul 28 11:04:27.522664 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.522625    2578 scope.go:117] "RemoveContainer" containerID="6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7"
Jul 28 11:04:27.531613 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.531592    2578 scope.go:117] "RemoveContainer" containerID="6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7"
Jul 28 11:04:27.531924 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:04:27.531904    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7\": container with ID starting with 6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7 not found: ID does not exist" containerID="6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7"
Jul 28 11:04:27.531969 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.531936    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7"} err="failed to get container status \"6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7\": rpc error: code = NotFound desc = could not find container \"6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7\": container with ID starting with 6d4e27706019240d6616da93a1063aafd7dc4ca7215655cd5651a84ae6fd6fc7 not found: ID does not exist"
Jul 28 11:04:27.539789 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.539749    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"]
Jul 28 11:04:27.543972 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:27.543943    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0b092d88-75779fdc6b-p6fg4"]
Jul 28 11:04:29.494573 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:29.494540    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d38c3c1e-9975-48d8-b81f-2a8b4c333911" path="/var/lib/kubelet/pods/d38c3c1e-9975-48d8-b81f-2a8b4c333911/volumes"
Jul 28 11:04:30.038989 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:30.038955    2578 scope.go:117] "RemoveContainer" containerID="3172106ecf725007ad0cc373a987cf7097c66212ebfa7dcdcb9324550ec2edf1"
Jul 28 11:04:39.892437 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:39.892404    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"]
Jul 28 11:04:54.912354 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:54.912303    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8" podUID="41a3a548-cd7c-4c29-abcc-78230ff10359" containerName="main" containerID="cri-o://421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05" gracePeriod=45
Jul 28 11:04:55.167509 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.167435    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:04:55.231385 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-model-cache\") pod \"41a3a548-cd7c-4c29-abcc-78230ff10359\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") "
Jul 28 11:04:55.231385 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfsk2\" (UniqueName: \"kubernetes.io/projected/41a3a548-cd7c-4c29-abcc-78230ff10359-kube-api-access-sfsk2\") pod \"41a3a548-cd7c-4c29-abcc-78230ff10359\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") "
Jul 28 11:04:55.231640 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231456    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a3a548-cd7c-4c29-abcc-78230ff10359-tls-certs\") pod \"41a3a548-cd7c-4c29-abcc-78230ff10359\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") "
Jul 28 11:04:55.231640 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-dshm\") pod \"41a3a548-cd7c-4c29-abcc-78230ff10359\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") "
Jul 28 11:04:55.231752 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-home\") pod \"41a3a548-cd7c-4c29-abcc-78230ff10359\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") "
Jul 28 11:04:55.231752 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231717    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-tmp-dir\") pod \"41a3a548-cd7c-4c29-abcc-78230ff10359\" (UID: \"41a3a548-cd7c-4c29-abcc-78230ff10359\") "
Jul 28 11:04:55.231850 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231646    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-model-cache" (OuterVolumeSpecName: "model-cache") pod "41a3a548-cd7c-4c29-abcc-78230ff10359" (UID: "41a3a548-cd7c-4c29-abcc-78230ff10359"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:55.231850 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.231843    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-home" (OuterVolumeSpecName: "home") pod "41a3a548-cd7c-4c29-abcc-78230ff10359" (UID: "41a3a548-cd7c-4c29-abcc-78230ff10359"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:55.232062 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.232040    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:55.232062 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.232048    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41a3a548-cd7c-4c29-abcc-78230ff10359" (UID: "41a3a548-cd7c-4c29-abcc-78230ff10359"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:55.232389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.232063    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:55.233875 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.233849    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41a3a548-cd7c-4c29-abcc-78230ff10359-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41a3a548-cd7c-4c29-abcc-78230ff10359" (UID: "41a3a548-cd7c-4c29-abcc-78230ff10359"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:04:55.233978 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.233878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-dshm" (OuterVolumeSpecName: "dshm") pod "41a3a548-cd7c-4c29-abcc-78230ff10359" (UID: "41a3a548-cd7c-4c29-abcc-78230ff10359"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:04:55.233978 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.233924    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41a3a548-cd7c-4c29-abcc-78230ff10359-kube-api-access-sfsk2" (OuterVolumeSpecName: "kube-api-access-sfsk2") pod "41a3a548-cd7c-4c29-abcc-78230ff10359" (UID: "41a3a548-cd7c-4c29-abcc-78230ff10359"). InnerVolumeSpecName "kube-api-access-sfsk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:04:55.332686 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.332643    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:55.332686 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.332676    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41a3a548-cd7c-4c29-abcc-78230ff10359-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:55.332686 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.332686    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfsk2\" (UniqueName: \"kubernetes.io/projected/41a3a548-cd7c-4c29-abcc-78230ff10359-kube-api-access-sfsk2\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:55.332686 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.332696    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41a3a548-cd7c-4c29-abcc-78230ff10359-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:04:55.659521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.659483    2578 generic.go:356] "Generic (PLEG): container finished" podID="41a3a548-cd7c-4c29-abcc-78230ff10359" containerID="421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05" exitCode=0
Jul 28 11:04:55.659705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.659528    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"
Jul 28 11:04:55.659705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.659559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8" event={"ID":"41a3a548-cd7c-4c29-abcc-78230ff10359","Type":"ContainerDied","Data":"421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05"}
Jul 28 11:04:55.659705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.659602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8" event={"ID":"41a3a548-cd7c-4c29-abcc-78230ff10359","Type":"ContainerDied","Data":"97db6c87b91e5c5e620d2abb3ab9892454e2ffa596c2f12e8b2c4f42e76293d5"}
Jul 28 11:04:55.659705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.659619    2578 scope.go:117] "RemoveContainer" containerID="421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05"
Jul 28 11:04:55.668472 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.668449    2578 scope.go:117] "RemoveContainer" containerID="421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05"
Jul 28 11:04:55.668819 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:04:55.668797    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05\": container with ID starting with 421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05 not found: ID does not exist" containerID="421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05"
Jul 28 11:04:55.668876 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.668828    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05"} err="failed to get container status \"421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05\": rpc error: code = NotFound desc = could not find container \"421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05\": container with ID starting with 421c9a321670ae0d9a271ed7493b6a01c01550668822e66cd890b33d323f9c05 not found: ID does not exist"
Jul 28 11:04:55.675405 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.675374    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"]
Jul 28 11:04:55.678632 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:55.678597    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-0b092d88/e2e-lifecycle-model-0b092d88-kserve-8685447755-lmvr8"]
Jul 28 11:04:57.494222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:04:57.494184    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41a3a548-cd7c-4c29-abcc-78230ff10359" path="/var/lib/kubelet/pods/41a3a548-cd7c-4c29-abcc-78230ff10359/volumes"
Jul 28 11:05:23.141473 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:23.141387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"]
Jul 28 11:05:23.142045 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:23.141605    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" containerID="cri-o://7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93" gracePeriod=30
Jul 28 11:05:24.416165 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:24.416127    2578 patch_prober.go:28] interesting pod/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 11:05:24.416595 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:24.416189    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 11:05:28.408358 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.408281    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:05:28.523038 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.522997    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thn5x\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-kube-api-access-thn5x\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-envoy\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523114    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istiod-ca-cert\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523133    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-credential-socket\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523154    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-token\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523170    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-podinfo\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523198    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-data\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-certs\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523648 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523282    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-socket\") pod \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\" (UID: \"96cdaf46-8bbb-41b1-b362-c9d6f75689f9\") "
Jul 28 11:05:28.523648 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523547    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:05:28.523755 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523687    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:05:28.523755 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.523722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:05:28.524206 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.524173    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:05:28.524502 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.524468    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-data" (OuterVolumeSpecName: "istio-data") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:05:28.525587 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.525564    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:05:28.525587 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.525574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-kube-api-access-thn5x" (OuterVolumeSpecName: "kube-api-access-thn5x") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "kube-api-access-thn5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:05:28.525714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.525619    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:05:28.525714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.525621    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-token" (OuterVolumeSpecName: "istio-token") pod "96cdaf46-8bbb-41b1-b362-c9d6f75689f9" (UID: "96cdaf46-8bbb-41b1-b362-c9d6f75689f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:05:28.624902 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624854    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.624902 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624898    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.624902 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624909    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.624902 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624918    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.624902 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624926    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.625224 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624933    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.625224 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624941    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.625224 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624949    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thn5x\" (UniqueName: \"kubernetes.io/projected/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-kube-api-access-thn5x\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.625224 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.624957    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cdaf46-8bbb-41b1-b362-c9d6f75689f9-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:05:28.821632 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.821595    2578 generic.go:356] "Generic (PLEG): container finished" podID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerID="7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93" exitCode=0
Jul 28 11:05:28.821816 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.821645    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"
Jul 28 11:05:28.821816 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.821678    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" event={"ID":"96cdaf46-8bbb-41b1-b362-c9d6f75689f9","Type":"ContainerDied","Data":"7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93"}
Jul 28 11:05:28.821816 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.821742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph" event={"ID":"96cdaf46-8bbb-41b1-b362-c9d6f75689f9","Type":"ContainerDied","Data":"febfa00975c8d68438899f045907e9d8a1aa5a7e1b90e6ada070a2333a204f4d"}
Jul 28 11:05:28.821816 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.821765    2578 scope.go:117] "RemoveContainer" containerID="7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93"
Jul 28 11:05:28.830952 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.830930    2578 scope.go:117] "RemoveContainer" containerID="56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"
Jul 28 11:05:28.839196 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.839174    2578 scope.go:117] "RemoveContainer" containerID="7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93"
Jul 28 11:05:28.839546 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:05:28.839524    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93\": container with ID starting with 7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93 not found: ID does not exist" containerID="7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93"
Jul 28 11:05:28.839628 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.839554    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93"} err="failed to get container status \"7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93\": rpc error: code = NotFound desc = could not find container \"7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93\": container with ID starting with 7d37467aed9e592894c6e64311dd793c2a0d1c3c9576a1360bcad1e6a2eeea93 not found: ID does not exist"
Jul 28 11:05:28.839628 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.839573    2578 scope.go:117] "RemoveContainer" containerID="56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"
Jul 28 11:05:28.839854 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:05:28.839835    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f\": container with ID starting with 56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f not found: ID does not exist" containerID="56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"
Jul 28 11:05:28.839899 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.839861    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f"} err="failed to get container status \"56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f\": rpc error: code = NotFound desc = could not find container \"56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f\": container with ID starting with 56ab3d4d0a1944eba0e08f15d83440f1285c0c7a25cec789e0fb29c896922e6f not found: ID does not exist"
Jul 28 11:05:28.844596 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.844560    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"]
Jul 28 11:05:28.848330 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:28.848299    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0b092d88-openshift-default-7dbc878dcb-9kzph"]
Jul 28 11:05:29.495072 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:29.495038    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" path="/var/lib/kubelet/pods/96cdaf46-8bbb-41b1-b362-c9d6f75689f9/volumes"
Jul 28 11:05:30.062689 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:05:30.062658    2578 scope.go:117] "RemoveContainer" containerID="c194c3068195852d631830a7e2074f1a5b11c89e8abe15af8bcfee856efc8bf5"
Jul 28 11:06:18.881690 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.881649    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"]
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882005    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d38c3c1e-9975-48d8-b81f-2a8b4c333911" containerName="payload-processing"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882018    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d38c3c1e-9975-48d8-b81f-2a8b4c333911" containerName="payload-processing"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882034    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882040    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882051    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41a3a548-cd7c-4c29-abcc-78230ff10359" containerName="main"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882060    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="41a3a548-cd7c-4c29-abcc-78230ff10359" containerName="main"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882073    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882080    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882139    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="41a3a548-cd7c-4c29-abcc-78230ff10359" containerName="main"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882148    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882154    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d38c3c1e-9975-48d8-b81f-2a8b4c333911" containerName="payload-processing"
Jul 28 11:06:18.882177 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.882161    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="96cdaf46-8bbb-41b1-b362-c9d6f75689f9" containerName="istio-proxy"
Jul 28 11:06:18.884903 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.884882    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.887323 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.887255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 11:06:18.887479 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.887384    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-730a1a-openshift-default-dockercfg-9gmsh\""
Jul 28 11:06:18.895322 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.895287    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"]
Jul 28 11:06:18.960453 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960412    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-credential-socket\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960666 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960475    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-token\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960666 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960501    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-podinfo\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960666 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960525    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-certs\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960666 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960595    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-data\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960694    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266ba9d7-7271-419f-9f68-d1e965555ff7-istiod-ca-cert\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xr2\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-kube-api-access-55xr2\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-socket\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:18.960897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:18.960855    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-envoy\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061479 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061443    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-envoy\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061479 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-credential-socket\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-token\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-podinfo\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-certs\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-data\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061965 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266ba9d7-7271-419f-9f68-d1e965555ff7-istiod-ca-cert\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061965 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061802    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xr2\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-kube-api-access-55xr2\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061965 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061843    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-socket\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.061965 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.061887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-credential-socket\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.062175 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.062064    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-data\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.062175 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.062089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-certs\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.062297 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.062223    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-socket\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.062505 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.062486    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266ba9d7-7271-419f-9f68-d1e965555ff7-istiod-ca-cert\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.064162 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.064140    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-envoy\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.064455 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.064436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-podinfo\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.078103 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.078062    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-token\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.078234 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.078111    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xr2\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-kube-api-access-55xr2\") pod \"e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") " pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.196653 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.196611    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:19.339316 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.339286    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"]
Jul 28 11:06:19.341886 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:06:19.341863    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266ba9d7_7271_419f_9f68_d1e965555ff7.slice/crio-6c8ee808bdeee67ba4263cdd08ebc64348cf190ab2bfd785e0797d139dc7a587 WatchSource:0}: Error finding container 6c8ee808bdeee67ba4263cdd08ebc64348cf190ab2bfd785e0797d139dc7a587: Status 404 returned error can't find the container with id 6c8ee808bdeee67ba4263cdd08ebc64348cf190ab2bfd785e0797d139dc7a587
Jul 28 11:06:19.343956 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.343918    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:06:19.344024 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.343985    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:06:19.344024 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:19.344011    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:06:20.056609 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:20.056572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" event={"ID":"266ba9d7-7271-419f-9f68-d1e965555ff7","Type":"ContainerStarted","Data":"826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"}
Jul 28 11:06:20.057021 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:20.056625    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" event={"ID":"266ba9d7-7271-419f-9f68-d1e965555ff7","Type":"ContainerStarted","Data":"6c8ee808bdeee67ba4263cdd08ebc64348cf190ab2bfd785e0797d139dc7a587"}
Jul 28 11:06:20.080185 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:20.080128    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podStartSLOduration=2.08011401 podStartE2EDuration="2.08011401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:20.07862607 +0000 UTC m=+2571.120718225" watchObservedRunningTime="2026-07-28 11:06:20.08011401 +0000 UTC m=+2571.122206163"
Jul 28 11:06:20.197842 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:20.197803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:20.202721 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:20.202692    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:21.061292 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:21.061249    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:21.062227 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:21.062206    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:22.883037 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.883002    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"]
Jul 28 11:06:22.886412 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.886392    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:22.888831 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.888807    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-730a1a-dockercfg-h8p42\""
Jul 28 11:06:22.888960 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.888835    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-730a1a\""
Jul 28 11:06:22.896997 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.896960    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"]
Jul 28 11:06:22.993340 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.993296    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1373c975-8b47-48ed-8d88-9f8045a1f2a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:22.993527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:22.993356    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h6gn\" (UniqueName: \"kubernetes.io/projected/1373c975-8b47-48ed-8d88-9f8045a1f2a2-kube-api-access-8h6gn\") pod \"payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:23.093888 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:23.093851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1373c975-8b47-48ed-8d88-9f8045a1f2a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:23.094088 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:23.093907    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h6gn\" (UniqueName: \"kubernetes.io/projected/1373c975-8b47-48ed-8d88-9f8045a1f2a2-kube-api-access-8h6gn\") pod \"payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:23.094689 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:23.094665    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1373c975-8b47-48ed-8d88-9f8045a1f2a2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:23.101941 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:23.101915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h6gn\" (UniqueName: \"kubernetes.io/projected/1373c975-8b47-48ed-8d88-9f8045a1f2a2-kube-api-access-8h6gn\") pod \"payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:23.198846 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:23.198800    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:23.357872 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:23.357842    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"]
Jul 28 11:06:23.360106 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:06:23.360070    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1373c975_8b47_48ed_8d88_9f8045a1f2a2.slice/crio-060537d5e281afa5e8d0f3b83b4d5d11bced4ced5a64d804f8c59a519e17c248 WatchSource:0}: Error finding container 060537d5e281afa5e8d0f3b83b4d5d11bced4ced5a64d804f8c59a519e17c248: Status 404 returned error can't find the container with id 060537d5e281afa5e8d0f3b83b4d5d11bced4ced5a64d804f8c59a519e17c248
Jul 28 11:06:24.076601 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:24.076559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz" event={"ID":"1373c975-8b47-48ed-8d88-9f8045a1f2a2","Type":"ContainerStarted","Data":"1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260"}
Jul 28 11:06:24.077111 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:24.076622    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz" event={"ID":"1373c975-8b47-48ed-8d88-9f8045a1f2a2","Type":"ContainerStarted","Data":"060537d5e281afa5e8d0f3b83b4d5d11bced4ced5a64d804f8c59a519e17c248"}
Jul 28 11:06:24.077111 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:24.076684    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:24.092494 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:24.092438    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz" podStartSLOduration=2.092420878 podStartE2EDuration="2.092420878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:24.091409502 +0000 UTC m=+2575.133501658" watchObservedRunningTime="2026-07-28 11:06:24.092420878 +0000 UTC m=+2575.134513031"
Jul 28 11:06:30.088764 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.088730    2578 scope.go:117] "RemoveContainer" containerID="b85e292c7cc0274bfda2648cb12e81de249f31ea96df1485853470a5897d76b1"
Jul 28 11:06:30.101554 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.101527    2578 scope.go:117] "RemoveContainer" containerID="53527846a59c011b5f131c4002ef1c2b7dcefafabe3ad604f487d27bf574f51d"
Jul 28 11:06:30.558682 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.558636    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"]
Jul 28 11:06:30.564083 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.564049    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.566793 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.566764    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2298f8\""
Jul 28 11:06:30.566965 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.566816    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2298f8-dockercfg-rg6j6\""
Jul 28 11:06:30.572879 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.572732    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"]
Jul 28 11:06:30.582169 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.582134    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"]
Jul 28 11:06:30.586369 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.586242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.596244 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.596214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"]
Jul 28 11:06:30.669242 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.669198    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsx9\" (UniqueName: \"kubernetes.io/projected/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-kube-api-access-zqsx9\") pod \"payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.669472 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.669382    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.669472 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.669459    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c72b2-6713-4ddb-b1ce-36502b10cd63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.669589 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.669489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr9kn\" (UniqueName: \"kubernetes.io/projected/d06c72b2-6713-4ddb-b1ce-36502b10cd63-kube-api-access-tr9kn\") pod \"payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.770171 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.770117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.770428 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.770211    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c72b2-6713-4ddb-b1ce-36502b10cd63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.770428 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.770316    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr9kn\" (UniqueName: \"kubernetes.io/projected/d06c72b2-6713-4ddb-b1ce-36502b10cd63-kube-api-access-tr9kn\") pod \"payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.770428 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.770389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsx9\" (UniqueName: \"kubernetes.io/projected/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-kube-api-access-zqsx9\") pod \"payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.771004 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.770981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c72b2-6713-4ddb-b1ce-36502b10cd63-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.771150 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.771128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.779448 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.779417    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr9kn\" (UniqueName: \"kubernetes.io/projected/d06c72b2-6713-4ddb-b1ce-36502b10cd63-kube-api-access-tr9kn\") pod \"payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:30.779599 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.779563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsx9\" (UniqueName: \"kubernetes.io/projected/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-kube-api-access-zqsx9\") pod \"payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.879567 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.879462    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:30.901308 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:30.901245    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:31.040938 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:31.040214    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"]
Jul 28 11:06:31.043536 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:06:31.043497    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3dd4ea7_a2e9_4261_a775_1c2f2fad643f.slice/crio-99b65c71107f07cb726ad067bcb3bb4fbb7d4cf25b476e36e6108bb34800262c WatchSource:0}: Error finding container 99b65c71107f07cb726ad067bcb3bb4fbb7d4cf25b476e36e6108bb34800262c: Status 404 returned error can't find the container with id 99b65c71107f07cb726ad067bcb3bb4fbb7d4cf25b476e36e6108bb34800262c
Jul 28 11:06:31.071075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:31.071044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"]
Jul 28 11:06:31.081547 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:06:31.081520    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06c72b2_6713_4ddb_b1ce_36502b10cd63.slice/crio-f779d66749c76256e334aa3251e0ccdba10338a3d0560c0a2ed49cbcd64787ee WatchSource:0}: Error finding container f779d66749c76256e334aa3251e0ccdba10338a3d0560c0a2ed49cbcd64787ee: Status 404 returned error can't find the container with id f779d66749c76256e334aa3251e0ccdba10338a3d0560c0a2ed49cbcd64787ee
Jul 28 11:06:31.112981 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:31.112947    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml" event={"ID":"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f","Type":"ContainerStarted","Data":"99b65c71107f07cb726ad067bcb3bb4fbb7d4cf25b476e36e6108bb34800262c"}
Jul 28 11:06:31.114463 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:31.114435    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc" event={"ID":"d06c72b2-6713-4ddb-b1ce-36502b10cd63","Type":"ContainerStarted","Data":"f779d66749c76256e334aa3251e0ccdba10338a3d0560c0a2ed49cbcd64787ee"}
Jul 28 11:06:32.122135 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:32.122089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc" event={"ID":"d06c72b2-6713-4ddb-b1ce-36502b10cd63","Type":"ContainerStarted","Data":"928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad"}
Jul 28 11:06:32.122725 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:32.122447    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:32.123948 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:32.123917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml" event={"ID":"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f","Type":"ContainerStarted","Data":"8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112"}
Jul 28 11:06:32.124062 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:32.123981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:32.140313 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:32.140233    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc" podStartSLOduration=2.14020827 podStartE2EDuration="2.14020827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:32.137782933 +0000 UTC m=+2583.179875086" watchObservedRunningTime="2026-07-28 11:06:32.14020827 +0000 UTC m=+2583.182300427"
Jul 28 11:06:32.153038 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:32.152977    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml" podStartSLOduration=2.152957519 podStartE2EDuration="2.152957519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:32.151805394 +0000 UTC m=+2583.193897573" watchObservedRunningTime="2026-07-28 11:06:32.152957519 +0000 UTC m=+2583.195049676"
Jul 28 11:06:33.131127 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:33.131029    2578 generic.go:356] "Generic (PLEG): container finished" podID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerID="826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125" exitCode=0
Jul 28 11:06:33.131127 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:33.131112    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" event={"ID":"266ba9d7-7271-419f-9f68-d1e965555ff7","Type":"ContainerDied","Data":"826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"}
Jul 28 11:06:33.131728 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:33.131695    2578 scope.go:117] "RemoveContainer" containerID="826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"
Jul 28 11:06:33.132664 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:33.132635    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:06:33.132727 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:33.132708    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:06:33.132771 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:33.132742    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:06:34.138359 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:34.138320    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" event={"ID":"266ba9d7-7271-419f-9f68-d1e965555ff7","Type":"ContainerStarted","Data":"554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35"}
Jul 28 11:06:34.197328 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:34.197290    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:34.198370 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:34.198346    2578 patch_prober.go:28] interesting pod/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 28 11:06:34.198441 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:34.198400    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 28 11:06:35.084522 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:35.084478    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:06:35.197961 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:35.197915    2578 patch_prober.go:28] interesting pod/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 28 11:06:35.198397 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:35.197988    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 28 11:06:36.061605 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:36.061555    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:36.197053 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:36.197014    2578 patch_prober.go:28] interesting pod/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 28 11:06:36.197240 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:36.197074    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 28 11:06:37.197415 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:37.197379    2578 patch_prober.go:28] interesting pod/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Jul 28 11:06:37.197886 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:37.197857    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Jul 28 11:06:38.201354 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:38.201323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:38.202213 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:38.202193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:06:43.133149 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:43.133114    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:06:43.133584 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:43.133190    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:06:56.353459 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.353374    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"]
Jul 28 11:06:56.357727 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.357704    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.361396 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.361328    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-730a1a\"/\"test-model-730a1a-kserve-self-signed-certs\""
Jul 28 11:06:56.361396 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.361380    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-730a1a\"/\"kube-root-ca.crt\""
Jul 28 11:06:56.361726 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.361649    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-730a1a\"/\"default-dockercfg-t8g64\""
Jul 28 11:06:56.361994 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.361927    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-730a1a\"/\"openshift-service-ca.crt\""
Jul 28 11:06:56.364663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.364637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"]
Jul 28 11:06:56.515017 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.514975    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-model-cache\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.515222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.515037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-home\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.515222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.515059    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-dshm\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.515222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.515074    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tls-certs\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.515222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.515104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tmp-dir\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.515222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.515129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kks2\" (UniqueName: \"kubernetes.io/projected/7b3ca67d-36ca-446d-a90e-1bcccac7182f-kube-api-access-8kks2\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616519 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616420    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-home\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616519 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616482    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-dshm\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616721 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tls-certs\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616721 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tmp-dir\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616721 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kks2\" (UniqueName: \"kubernetes.io/projected/7b3ca67d-36ca-446d-a90e-1bcccac7182f-kube-api-access-8kks2\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616871 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-model-cache\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616925 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-home\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.616925 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.616910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tmp-dir\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.617072 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.617053    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-model-cache\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.619014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.618994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-dshm\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.619257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.619241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tls-certs\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.624905 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.624878    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kks2\" (UniqueName: \"kubernetes.io/projected/7b3ca67d-36ca-446d-a90e-1bcccac7182f-kube-api-access-8kks2\") pod \"test-model-730a1a-kserve-57bf65d66c-xtps6\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") " pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.669143 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.669102    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:06:56.810073 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:56.810006    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"]
Jul 28 11:06:56.812221 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:06:56.812194    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b3ca67d_36ca_446d_a90e_1bcccac7182f.slice/crio-505fe606e8aee4b8e5b0d1f19b9cb58a6d193b19cd2dca767352edcaa266f007 WatchSource:0}: Error finding container 505fe606e8aee4b8e5b0d1f19b9cb58a6d193b19cd2dca767352edcaa266f007: Status 404 returned error can't find the container with id 505fe606e8aee4b8e5b0d1f19b9cb58a6d193b19cd2dca767352edcaa266f007
Jul 28 11:06:57.248962 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:57.248924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6" event={"ID":"7b3ca67d-36ca-446d-a90e-1bcccac7182f","Type":"ContainerStarted","Data":"eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc"}
Jul 28 11:06:57.248962 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:57.248969    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6" event={"ID":"7b3ca67d-36ca-446d-a90e-1bcccac7182f","Type":"ContainerStarted","Data":"505fe606e8aee4b8e5b0d1f19b9cb58a6d193b19cd2dca767352edcaa266f007"}
Jul 28 11:06:57.267057 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:06:57.267002    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6" podStartSLOduration=1.2669857279999999 podStartE2EDuration="1.266985728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:06:57.264637827 +0000 UTC m=+2608.306729980" watchObservedRunningTime="2026-07-28 11:06:57.266985728 +0000 UTC m=+2608.309077928"
Jul 28 11:07:06.669499 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:07:06.669458    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:07:06.669499 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:07:06.669510    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:07:06.682336 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:07:06.682304    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:07:07.311909 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:07:07.311879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:08:15.144005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:15.143952    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"]
Jul 28 11:08:26.664076 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:26.663981    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"]
Jul 28 11:08:26.664531 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:26.664252    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz" podUID="1373c975-8b47-48ed-8d88-9f8045a1f2a2" containerName="payload-processing" containerID="cri-o://1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260" gracePeriod=30
Jul 28 11:08:26.912907 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:26.912871    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:08:27.042026 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.041988    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1373c975-8b47-48ed-8d88-9f8045a1f2a2-plugins-config-volume\") pod \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") "
Jul 28 11:08:27.042026 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.042035    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h6gn\" (UniqueName: \"kubernetes.io/projected/1373c975-8b47-48ed-8d88-9f8045a1f2a2-kube-api-access-8h6gn\") pod \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\" (UID: \"1373c975-8b47-48ed-8d88-9f8045a1f2a2\") "
Jul 28 11:08:27.042518 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.042486    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1373c975-8b47-48ed-8d88-9f8045a1f2a2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1373c975-8b47-48ed-8d88-9f8045a1f2a2" (UID: "1373c975-8b47-48ed-8d88-9f8045a1f2a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:08:27.044340 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.044314    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1373c975-8b47-48ed-8d88-9f8045a1f2a2-kube-api-access-8h6gn" (OuterVolumeSpecName: "kube-api-access-8h6gn") pod "1373c975-8b47-48ed-8d88-9f8045a1f2a2" (UID: "1373c975-8b47-48ed-8d88-9f8045a1f2a2"). InnerVolumeSpecName "kube-api-access-8h6gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:27.142958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.142914    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1373c975-8b47-48ed-8d88-9f8045a1f2a2-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:27.142958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.142948    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h6gn\" (UniqueName: \"kubernetes.io/projected/1373c975-8b47-48ed-8d88-9f8045a1f2a2-kube-api-access-8h6gn\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:27.690372 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.690337    2578 generic.go:356] "Generic (PLEG): container finished" podID="1373c975-8b47-48ed-8d88-9f8045a1f2a2" containerID="1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260" exitCode=0
Jul 28 11:08:27.690824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.690381    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"
Jul 28 11:08:27.690824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.690389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz" event={"ID":"1373c975-8b47-48ed-8d88-9f8045a1f2a2","Type":"ContainerDied","Data":"1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260"}
Jul 28 11:08:27.690824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.690430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz" event={"ID":"1373c975-8b47-48ed-8d88-9f8045a1f2a2","Type":"ContainerDied","Data":"060537d5e281afa5e8d0f3b83b4d5d11bced4ced5a64d804f8c59a519e17c248"}
Jul 28 11:08:27.690824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.690452    2578 scope.go:117] "RemoveContainer" containerID="1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260"
Jul 28 11:08:27.699167 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.699144    2578 scope.go:117] "RemoveContainer" containerID="1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260"
Jul 28 11:08:27.699517 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:08:27.699487    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260\": container with ID starting with 1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260 not found: ID does not exist" containerID="1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260"
Jul 28 11:08:27.699650 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.699522    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260"} err="failed to get container status \"1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260\": rpc error: code = NotFound desc = could not find container \"1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260\": container with ID starting with 1851820e32030a6c9b90bba653694ba1d1133c030332625ffef9c3329aa74260 not found: ID does not exist"
Jul 28 11:08:27.708746 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.708713    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"]
Jul 28 11:08:27.712614 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:27.712587    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-730a1a-6f88749588-phhlz"]
Jul 28 11:08:29.495573 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:29.495533    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1373c975-8b47-48ed-8d88-9f8045a1f2a2" path="/var/lib/kubelet/pods/1373c975-8b47-48ed-8d88-9f8045a1f2a2/volumes"
Jul 28 11:08:30.002097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.002066    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:08:30.012828 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.012799    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:08:30.164623 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.164559    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6" podUID="7b3ca67d-36ca-446d-a90e-1bcccac7182f" containerName="main" containerID="cri-o://eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc" gracePeriod=45
Jul 28 11:08:30.423880 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.423845    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:08:30.573993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.573905    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-home\") pod \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") "
Jul 28 11:08:30.573993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.573951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kks2\" (UniqueName: \"kubernetes.io/projected/7b3ca67d-36ca-446d-a90e-1bcccac7182f-kube-api-access-8kks2\") pod \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") "
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tls-certs\") pod \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") "
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574026    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-model-cache\") pod \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") "
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574064    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tmp-dir\") pod \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") "
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574119    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-dshm\") pod \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\" (UID: \"7b3ca67d-36ca-446d-a90e-1bcccac7182f\") "
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574227    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-home" (OuterVolumeSpecName: "home") pod "7b3ca67d-36ca-446d-a90e-1bcccac7182f" (UID: "7b3ca67d-36ca-446d-a90e-1bcccac7182f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b3ca67d-36ca-446d-a90e-1bcccac7182f" (UID: "7b3ca67d-36ca-446d-a90e-1bcccac7182f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:30.574521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b3ca67d-36ca-446d-a90e-1bcccac7182f" (UID: "7b3ca67d-36ca-446d-a90e-1bcccac7182f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:30.574851 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574570    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:30.574851 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574589    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:30.574851 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.574616    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:30.576678 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.576644    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-dshm" (OuterVolumeSpecName: "dshm") pod "7b3ca67d-36ca-446d-a90e-1bcccac7182f" (UID: "7b3ca67d-36ca-446d-a90e-1bcccac7182f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:30.576799 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.576700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b3ca67d-36ca-446d-a90e-1bcccac7182f" (UID: "7b3ca67d-36ca-446d-a90e-1bcccac7182f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:08:30.576799 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.576786    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3ca67d-36ca-446d-a90e-1bcccac7182f-kube-api-access-8kks2" (OuterVolumeSpecName: "kube-api-access-8kks2") pod "7b3ca67d-36ca-446d-a90e-1bcccac7182f" (UID: "7b3ca67d-36ca-446d-a90e-1bcccac7182f"). InnerVolumeSpecName "kube-api-access-8kks2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:30.675257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.675221    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kks2\" (UniqueName: \"kubernetes.io/projected/7b3ca67d-36ca-446d-a90e-1bcccac7182f-kube-api-access-8kks2\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:30.675257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.675254    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3ca67d-36ca-446d-a90e-1bcccac7182f-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:30.675257 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.675295    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b3ca67d-36ca-446d-a90e-1bcccac7182f-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:30.709399 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.709361    2578 generic.go:356] "Generic (PLEG): container finished" podID="7b3ca67d-36ca-446d-a90e-1bcccac7182f" containerID="eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc" exitCode=0
Jul 28 11:08:30.709597 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.709410    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"
Jul 28 11:08:30.709597 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.709426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6" event={"ID":"7b3ca67d-36ca-446d-a90e-1bcccac7182f","Type":"ContainerDied","Data":"eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc"}
Jul 28 11:08:30.709597 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.709458    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6" event={"ID":"7b3ca67d-36ca-446d-a90e-1bcccac7182f","Type":"ContainerDied","Data":"505fe606e8aee4b8e5b0d1f19b9cb58a6d193b19cd2dca767352edcaa266f007"}
Jul 28 11:08:30.709597 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.709474    2578 scope.go:117] "RemoveContainer" containerID="eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc"
Jul 28 11:08:30.719111 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.719088    2578 scope.go:117] "RemoveContainer" containerID="eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc"
Jul 28 11:08:30.719464 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:08:30.719440    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc\": container with ID starting with eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc not found: ID does not exist" containerID="eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc"
Jul 28 11:08:30.719556 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.719474    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc"} err="failed to get container status \"eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc\": rpc error: code = NotFound desc = could not find container \"eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc\": container with ID starting with eccadb00719ac561715460dd926edc1adbab659359a50924eb9f55d793fea9cc not found: ID does not exist"
Jul 28 11:08:30.729418 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.729382    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"]
Jul 28 11:08:30.735188 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:30.735152    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-730a1a/test-model-730a1a-kserve-57bf65d66c-xtps6"]
Jul 28 11:08:31.494957 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:31.494919    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b3ca67d-36ca-446d-a90e-1bcccac7182f" path="/var/lib/kubelet/pods/7b3ca67d-36ca-446d-a90e-1bcccac7182f/volumes"
Jul 28 11:08:37.783598 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:37.783558    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"]
Jul 28 11:08:37.784029 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:37.783826    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy" containerID="cri-o://554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35" gracePeriod=30
Jul 28 11:08:38.204678 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:38.204630    2578 patch_prober.go:28] interesting pod/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 11:08:38.204859 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:38.204703    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 11:08:43.042533 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.042499    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:08:43.080901 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080794    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-envoy\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.080901 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080848    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55xr2\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-kube-api-access-55xr2\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.080901 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080874    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-podinfo\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.080901 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266ba9d7-7271-419f-9f68-d1e965555ff7-istiod-ca-cert\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.081259 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080917    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-certs\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.081259 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080945    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-data\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.081259 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.080999    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-socket\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.081259 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.081029    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-token\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.081259 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.081071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-credential-socket\") pod \"266ba9d7-7271-419f-9f68-d1e965555ff7\" (UID: \"266ba9d7-7271-419f-9f68-d1e965555ff7\") "
Jul 28 11:08:43.081563 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.081262    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:43.081563 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.081390    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266ba9d7-7271-419f-9f68-d1e965555ff7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:08:43.081563 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.081401    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:43.081563 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.081529    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:43.082092 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.082072    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-data" (OuterVolumeSpecName: "istio-data") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:43.084090 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.084047    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:08:43.084090 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.084061    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-token" (OuterVolumeSpecName: "istio-token") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:43.084090 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.084083    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:08:43.084305 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.084095    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-kube-api-access-55xr2" (OuterVolumeSpecName: "kube-api-access-55xr2") pod "266ba9d7-7271-419f-9f68-d1e965555ff7" (UID: "266ba9d7-7271-419f-9f68-d1e965555ff7"). InnerVolumeSpecName "kube-api-access-55xr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:43.182043 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.181997    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/266ba9d7-7271-419f-9f68-d1e965555ff7-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182043 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182035    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182043 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182044    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182043 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182052    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182060    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182068    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182076    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182084    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xr2\" (UniqueName: \"kubernetes.io/projected/266ba9d7-7271-419f-9f68-d1e965555ff7-kube-api-access-55xr2\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.182374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.182094    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/266ba9d7-7271-419f-9f68-d1e965555ff7-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:43.773045 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.772936    2578 generic.go:356] "Generic (PLEG): container finished" podID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerID="554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35" exitCode=0
Jul 28 11:08:43.773045 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.772992    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"
Jul 28 11:08:43.773045 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.773025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" event={"ID":"266ba9d7-7271-419f-9f68-d1e965555ff7","Type":"ContainerDied","Data":"554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35"}
Jul 28 11:08:43.773386 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.773082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh" event={"ID":"266ba9d7-7271-419f-9f68-d1e965555ff7","Type":"ContainerDied","Data":"6c8ee808bdeee67ba4263cdd08ebc64348cf190ab2bfd785e0797d139dc7a587"}
Jul 28 11:08:43.773386 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.773103    2578 scope.go:117] "RemoveContainer" containerID="554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35"
Jul 28 11:08:43.782082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.782060    2578 scope.go:117] "RemoveContainer" containerID="826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"
Jul 28 11:08:43.791241 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.791208    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"]
Jul 28 11:08:43.791241 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.791239    2578 scope.go:117] "RemoveContainer" containerID="554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35"
Jul 28 11:08:43.791634 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:08:43.791610    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35\": container with ID starting with 554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35 not found: ID does not exist" containerID="554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35"
Jul 28 11:08:43.791693 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.791642    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35"} err="failed to get container status \"554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35\": rpc error: code = NotFound desc = could not find container \"554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35\": container with ID starting with 554a9a4650dbb04c8cc937ddd62f64c54f5c65f2f973ec91f6092426933aac35 not found: ID does not exist"
Jul 28 11:08:43.791693 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.791660    2578 scope.go:117] "RemoveContainer" containerID="826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"
Jul 28 11:08:43.791945 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:08:43.791928    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125\": container with ID starting with 826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125 not found: ID does not exist" containerID="826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"
Jul 28 11:08:43.791995 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.791949    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125"} err="failed to get container status \"826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125\": rpc error: code = NotFound desc = could not find container \"826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125\": container with ID starting with 826b7888d2529ef68b28ec512375b2cc8c2e002bff17e66bfd4efbc87c6e8125 not found: ID does not exist"
Jul 28 11:08:43.794719 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:43.794691    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-730a1a-openshift-default-78b6d559fb-6szsh"]
Jul 28 11:08:45.494954 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:45.494916    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" path="/var/lib/kubelet/pods/266ba9d7-7271-419f-9f68-d1e965555ff7/volumes"
Jul 28 11:08:49.014036 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.013995    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"]
Jul 28 11:08:49.014695 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.014217    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc" podUID="d06c72b2-6713-4ddb-b1ce-36502b10cd63" containerName="payload-processing" containerID="cri-o://928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad" gracePeriod=30
Jul 28 11:08:49.019375 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.019263    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"]
Jul 28 11:08:49.019758 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.019579    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml" podUID="f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" containerName="payload-pre-processing" containerID="cri-o://8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112" gracePeriod=30
Jul 28 11:08:49.292883 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.292854    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:08:49.295993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.295952    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:08:49.328746 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.328708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c72b2-6713-4ddb-b1ce-36502b10cd63-plugins-config-volume\") pod \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") "
Jul 28 11:08:49.328919 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.328814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqsx9\" (UniqueName: \"kubernetes.io/projected/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-kube-api-access-zqsx9\") pod \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") "
Jul 28 11:08:49.328997 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.328922    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr9kn\" (UniqueName: \"kubernetes.io/projected/d06c72b2-6713-4ddb-b1ce-36502b10cd63-kube-api-access-tr9kn\") pod \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\" (UID: \"d06c72b2-6713-4ddb-b1ce-36502b10cd63\") "
Jul 28 11:08:49.328997 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.328969    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-plugins-config-volume\") pod \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\" (UID: \"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f\") "
Jul 28 11:08:49.329192 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.329168    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d06c72b2-6713-4ddb-b1ce-36502b10cd63-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d06c72b2-6713-4ddb-b1ce-36502b10cd63" (UID: "d06c72b2-6713-4ddb-b1ce-36502b10cd63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:08:49.329507 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.329474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" (UID: "f3dd4ea7-a2e9-4261-a775-1c2f2fad643f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:08:49.331361 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.331331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d06c72b2-6713-4ddb-b1ce-36502b10cd63-kube-api-access-tr9kn" (OuterVolumeSpecName: "kube-api-access-tr9kn") pod "d06c72b2-6713-4ddb-b1ce-36502b10cd63" (UID: "d06c72b2-6713-4ddb-b1ce-36502b10cd63"). InnerVolumeSpecName "kube-api-access-tr9kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:49.331461 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.331331    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-kube-api-access-zqsx9" (OuterVolumeSpecName: "kube-api-access-zqsx9") pod "f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" (UID: "f3dd4ea7-a2e9-4261-a775-1c2f2fad643f"). InnerVolumeSpecName "kube-api-access-zqsx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:08:49.429817 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.429771    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d06c72b2-6713-4ddb-b1ce-36502b10cd63-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:49.429817 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.429807    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqsx9\" (UniqueName: \"kubernetes.io/projected/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-kube-api-access-zqsx9\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:49.429817 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.429818    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr9kn\" (UniqueName: \"kubernetes.io/projected/d06c72b2-6713-4ddb-b1ce-36502b10cd63-kube-api-access-tr9kn\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:49.429817 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.429826    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:08:49.803584 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.803545    2578 generic.go:356] "Generic (PLEG): container finished" podID="d06c72b2-6713-4ddb-b1ce-36502b10cd63" containerID="928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad" exitCode=0
Jul 28 11:08:49.803789 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.803590    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"
Jul 28 11:08:49.803789 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.803631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc" event={"ID":"d06c72b2-6713-4ddb-b1ce-36502b10cd63","Type":"ContainerDied","Data":"928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad"}
Jul 28 11:08:49.803789 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.803685    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc" event={"ID":"d06c72b2-6713-4ddb-b1ce-36502b10cd63","Type":"ContainerDied","Data":"f779d66749c76256e334aa3251e0ccdba10338a3d0560c0a2ed49cbcd64787ee"}
Jul 28 11:08:49.803789 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.803711    2578 scope.go:117] "RemoveContainer" containerID="928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad"
Jul 28 11:08:49.805148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.805113    2578 generic.go:356] "Generic (PLEG): container finished" podID="f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" containerID="8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112" exitCode=0
Jul 28 11:08:49.805232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.805152    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml" event={"ID":"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f","Type":"ContainerDied","Data":"8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112"}
Jul 28 11:08:49.805232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.805172    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"
Jul 28 11:08:49.805232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.805188    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml" event={"ID":"f3dd4ea7-a2e9-4261-a775-1c2f2fad643f","Type":"ContainerDied","Data":"99b65c71107f07cb726ad067bcb3bb4fbb7d4cf25b476e36e6108bb34800262c"}
Jul 28 11:08:49.812914 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.812890    2578 scope.go:117] "RemoveContainer" containerID="928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad"
Jul 28 11:08:49.813235 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:08:49.813213    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad\": container with ID starting with 928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad not found: ID does not exist" containerID="928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad"
Jul 28 11:08:49.813361 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.813250    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad"} err="failed to get container status \"928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad\": rpc error: code = NotFound desc = could not find container \"928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad\": container with ID starting with 928edf098b88bae477c9077e91b20ea70c650eba057fe38b71d73284402ed9ad not found: ID does not exist"
Jul 28 11:08:49.813361 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.813303    2578 scope.go:117] "RemoveContainer" containerID="8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112"
Jul 28 11:08:49.821575 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.821543    2578 scope.go:117] "RemoveContainer" containerID="8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112"
Jul 28 11:08:49.822064 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:08:49.822020    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112\": container with ID starting with 8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112 not found: ID does not exist" containerID="8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112"
Jul 28 11:08:49.822190 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.822062    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112"} err="failed to get container status \"8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112\": rpc error: code = NotFound desc = could not find container \"8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112\": container with ID starting with 8f8c666494fae1df7f9e1c3490d9d6afde1ba22886e48dba4fda0d86dd40b112 not found: ID does not exist"
Jul 28 11:08:49.823771 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.823749    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"]
Jul 28 11:08:49.827484 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.827459    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2298f8-5d6f8b5bb4-57vml"]
Jul 28 11:08:49.837154 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.837119    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"]
Jul 28 11:08:49.840607 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:49.840575    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2298f8-8459d65dc8-nkmvc"]
Jul 28 11:08:51.494550 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:51.494513    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d06c72b2-6713-4ddb-b1ce-36502b10cd63" path="/var/lib/kubelet/pods/d06c72b2-6713-4ddb-b1ce-36502b10cd63/volumes"
Jul 28 11:08:51.494937 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:08:51.494876    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" path="/var/lib/kubelet/pods/f3dd4ea7-a2e9-4261-a775-1c2f2fad643f/volumes"
Jul 28 11:09:00.751480 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751443    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"]
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751797    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1373c975-8b47-48ed-8d88-9f8045a1f2a2" containerName="payload-processing"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751810    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1373c975-8b47-48ed-8d88-9f8045a1f2a2" containerName="payload-processing"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751823    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751829    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751836    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b3ca67d-36ca-446d-a90e-1bcccac7182f" containerName="main"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751842    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b3ca67d-36ca-446d-a90e-1bcccac7182f" containerName="main"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751851    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d06c72b2-6713-4ddb-b1ce-36502b10cd63" containerName="payload-processing"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751856    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d06c72b2-6713-4ddb-b1ce-36502b10cd63" containerName="payload-processing"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751866    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" containerName="payload-pre-processing"
Jul 28 11:09:00.751870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751871    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" containerName="payload-pre-processing"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751885    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751891    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751944    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b3ca67d-36ca-446d-a90e-1bcccac7182f" containerName="main"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751951    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751955    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1373c975-8b47-48ed-8d88-9f8045a1f2a2" containerName="payload-processing"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751961    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3dd4ea7-a2e9-4261-a775-1c2f2fad643f" containerName="payload-pre-processing"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751969    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="266ba9d7-7271-419f-9f68-d1e965555ff7" containerName="istio-proxy"
Jul 28 11:09:00.752172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.751975    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d06c72b2-6713-4ddb-b1ce-36502b10cd63" containerName="payload-processing"
Jul 28 11:09:00.756882 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.756859    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.759405 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.759379    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 11:09:00.759405 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.759402    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1912e6-openshift-default-dockercfg-tbfs8\""
Jul 28 11:09:00.766280 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.766244    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"]
Jul 28 11:09:00.823079 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-data\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823312 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823126    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5dd\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-kube-api-access-6p5dd\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823312 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823192    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-envoy\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823312 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-token\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823312 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823230    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istiod-ca-cert\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823312 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823249    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-socket\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823530 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823321    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-certs\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823530 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823379    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-credential-socket\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.823530 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.823417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-podinfo\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924050 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-data\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924050 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5dd\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-kube-api-access-6p5dd\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924418 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-envoy\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924418 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-token\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924418 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istiod-ca-cert\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924418 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-socket\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-certs\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-credential-socket\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924479    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-data\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924530    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-podinfo\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924610    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-socket\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924894 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-credential-socket\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.924894 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.924796    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-certs\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.925054 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.925034    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istiod-ca-cert\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.926668 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.926647    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-envoy\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.927124 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.927107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-podinfo\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.932598 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.932569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-token\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:00.932718 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:00.932625    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5dd\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-kube-api-access-6p5dd\") pod \"e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") " pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:01.068318 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.068198    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:01.216396 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.216365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"]
Jul 28 11:09:01.218573 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:09:01.218546    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f21d2e6_f74d_4071_9a02_5969cb8e4f0a.slice/crio-234c4e8c81cac425d105b7679d785d731723f123cd0bc44de16b3f5ce4aa8abe WatchSource:0}: Error finding container 234c4e8c81cac425d105b7679d785d731723f123cd0bc44de16b3f5ce4aa8abe: Status 404 returned error can't find the container with id 234c4e8c81cac425d105b7679d785d731723f123cd0bc44de16b3f5ce4aa8abe
Jul 28 11:09:01.221220 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.221172    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:09:01.221353 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.221252    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:09:01.221353 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.221315    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:09:01.868234 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.868195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" event={"ID":"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a","Type":"ContainerStarted","Data":"0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"}
Jul 28 11:09:01.868660 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.868255    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" event={"ID":"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a","Type":"ContainerStarted","Data":"234c4e8c81cac425d105b7679d785d731723f123cd0bc44de16b3f5ce4aa8abe"}
Jul 28 11:09:01.889594 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:01.889536    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podStartSLOduration=1.889516921 podStartE2EDuration="1.889516921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:01.888049028 +0000 UTC m=+2732.930141182" watchObservedRunningTime="2026-07-28 11:09:01.889516921 +0000 UTC m=+2732.931609144"
Jul 28 11:09:02.068882 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:02.068839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:02.074213 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:02.074181    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:02.873987 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:02.873958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:02.875075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:02.875058    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:04.894314 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.894261    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"]
Jul 28 11:09:04.897864 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.897841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:04.900439 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.900405    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-1912e6-dockercfg-clbh9\""
Jul 28 11:09:04.900439 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.900404    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-1912e6\""
Jul 28 11:09:04.910023 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.909992    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"]
Jul 28 11:09:04.963993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.963952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6cc156-f924-4f66-8460-c522fd30fc59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") " pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:04.964251 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:04.964161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dzdb\" (UniqueName: \"kubernetes.io/projected/da6cc156-f924-4f66-8460-c522fd30fc59-kube-api-access-9dzdb\") pod \"payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") " pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.065154 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.065121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dzdb\" (UniqueName: \"kubernetes.io/projected/da6cc156-f924-4f66-8460-c522fd30fc59-kube-api-access-9dzdb\") pod \"payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") " pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.065365 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.065178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6cc156-f924-4f66-8460-c522fd30fc59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") " pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.066092 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.066069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6cc156-f924-4f66-8460-c522fd30fc59-plugins-config-volume\") pod \"payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") " pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.073461 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.073423    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dzdb\" (UniqueName: \"kubernetes.io/projected/da6cc156-f924-4f66-8460-c522fd30fc59-kube-api-access-9dzdb\") pod \"payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") " pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.209433 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.209384    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.352111 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.352084    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"]
Jul 28 11:09:05.354449 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:09:05.354414    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6cc156_f924_4f66_8460_c522fd30fc59.slice/crio-417ffe92c817b0bb652d539e5a393627fe0bc2296a84f3d0c8d1a18a156a5dbc WatchSource:0}: Error finding container 417ffe92c817b0bb652d539e5a393627fe0bc2296a84f3d0c8d1a18a156a5dbc: Status 404 returned error can't find the container with id 417ffe92c817b0bb652d539e5a393627fe0bc2296a84f3d0c8d1a18a156a5dbc
Jul 28 11:09:05.892566 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.892514    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz" event={"ID":"da6cc156-f924-4f66-8460-c522fd30fc59","Type":"ContainerStarted","Data":"47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c"}
Jul 28 11:09:05.892566 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.892572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz" event={"ID":"da6cc156-f924-4f66-8460-c522fd30fc59","Type":"ContainerStarted","Data":"417ffe92c817b0bb652d539e5a393627fe0bc2296a84f3d0c8d1a18a156a5dbc"}
Jul 28 11:09:05.892840 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.892620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:05.910687 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:05.910618    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz" podStartSLOduration=1.910587081 podStartE2EDuration="1.910587081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:05.909810676 +0000 UTC m=+2736.951902844" watchObservedRunningTime="2026-07-28 11:09:05.910587081 +0000 UTC m=+2736.952679236"
Jul 28 11:09:13.411907 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.411857    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"]
Jul 28 11:09:13.429291 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.429224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"]
Jul 28 11:09:13.429482 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.429407    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.432661 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.432626    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-86170f-dockercfg-sqphq\""
Jul 28 11:09:13.432841 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.432626    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-86170f\""
Jul 28 11:09:13.439755 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.439724    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"]
Jul 28 11:09:13.443839 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.443816    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.456572 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.456539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"]
Jul 28 11:09:13.550884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.550843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jqq7\" (UniqueName: \"kubernetes.io/projected/cf65c864-692c-4605-8eb6-f539ac6ac400-kube-api-access-5jqq7\") pod \"payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.551073 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.550916    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e95252-5de3-4dcf-a48a-e752766529d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.551073 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.550965    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8b85\" (UniqueName: \"kubernetes.io/projected/16e95252-5de3-4dcf-a48a-e752766529d4-kube-api-access-t8b85\") pod \"payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.551073 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.550991    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65c864-692c-4605-8eb6-f539ac6ac400-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.652103 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.652070    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8b85\" (UniqueName: \"kubernetes.io/projected/16e95252-5de3-4dcf-a48a-e752766529d4-kube-api-access-t8b85\") pod \"payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.652311 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.652121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65c864-692c-4605-8eb6-f539ac6ac400-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.652311 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.652157    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jqq7\" (UniqueName: \"kubernetes.io/projected/cf65c864-692c-4605-8eb6-f539ac6ac400-kube-api-access-5jqq7\") pod \"payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.652436 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.652328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e95252-5de3-4dcf-a48a-e752766529d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.652991 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.652972    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65c864-692c-4605-8eb6-f539ac6ac400-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.653032 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.652981    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e95252-5de3-4dcf-a48a-e752766529d4-plugins-config-volume\") pod \"payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.661233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.661202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jqq7\" (UniqueName: \"kubernetes.io/projected/cf65c864-692c-4605-8eb6-f539ac6ac400-kube-api-access-5jqq7\") pod \"payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.661410 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.661215    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8b85\" (UniqueName: \"kubernetes.io/projected/16e95252-5de3-4dcf-a48a-e752766529d4-kube-api-access-t8b85\") pod \"payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") " pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.743227 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.743192    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:13.756065 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.756039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:13.915701 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.913819    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"]
Jul 28 11:09:13.918231 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:09:13.918199    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf65c864_692c_4605_8eb6_f539ac6ac400.slice/crio-16fc6ecee7882ac18a0985a22429ddd1b84d3f35a2784e3f137b1239ba86d80a WatchSource:0}: Error finding container 16fc6ecee7882ac18a0985a22429ddd1b84d3f35a2784e3f137b1239ba86d80a: Status 404 returned error can't find the container with id 16fc6ecee7882ac18a0985a22429ddd1b84d3f35a2784e3f137b1239ba86d80a
Jul 28 11:09:13.941196 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.941153    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"]
Jul 28 11:09:13.944150 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:13.944122    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl" event={"ID":"cf65c864-692c-4605-8eb6-f539ac6ac400","Type":"ContainerStarted","Data":"16fc6ecee7882ac18a0985a22429ddd1b84d3f35a2784e3f137b1239ba86d80a"}
Jul 28 11:09:13.953894 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:09:13.953861    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e95252_5de3_4dcf_a48a_e752766529d4.slice/crio-e31c291e977d5c1beee43fdbe062111082417bc865ea1134b27fb83fda2929ee WatchSource:0}: Error finding container e31c291e977d5c1beee43fdbe062111082417bc865ea1134b27fb83fda2929ee: Status 404 returned error can't find the container with id e31c291e977d5c1beee43fdbe062111082417bc865ea1134b27fb83fda2929ee
Jul 28 11:09:14.951731 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.951688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl" event={"ID":"cf65c864-692c-4605-8eb6-f539ac6ac400","Type":"ContainerStarted","Data":"cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e"}
Jul 28 11:09:14.952216 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.951823    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:14.953521 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.953490    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b" event={"ID":"16e95252-5de3-4dcf-a48a-e752766529d4","Type":"ContainerStarted","Data":"a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999"}
Jul 28 11:09:14.953672 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.953549    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b" event={"ID":"16e95252-5de3-4dcf-a48a-e752766529d4","Type":"ContainerStarted","Data":"e31c291e977d5c1beee43fdbe062111082417bc865ea1134b27fb83fda2929ee"}
Jul 28 11:09:14.953672 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.953590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:14.968375 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.968319    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl" podStartSLOduration=1.968301566 podStartE2EDuration="1.968301566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:14.967459175 +0000 UTC m=+2746.009551343" watchObservedRunningTime="2026-07-28 11:09:14.968301566 +0000 UTC m=+2746.010393727"
Jul 28 11:09:14.983871 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:14.983811    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b" podStartSLOduration=1.983793601 podStartE2EDuration="1.983793601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:09:14.982299871 +0000 UTC m=+2746.024392027" watchObservedRunningTime="2026-07-28 11:09:14.983793601 +0000 UTC m=+2746.025885753"
Jul 28 11:09:16.900703 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.900605    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:16.965729 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.965692    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerID="0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e" exitCode=0
Jul 28 11:09:16.965920 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.965764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" event={"ID":"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a","Type":"ContainerDied","Data":"0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"}
Jul 28 11:09:16.966175 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.966158    2578 scope.go:117] "RemoveContainer" containerID="0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"
Jul 28 11:09:16.967213 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.967155    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:09:16.967314 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.967255    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:09:16.967359 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:16.967319    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:09:17.874346 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:17.874305    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:17.972744 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:17.972705    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" event={"ID":"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a","Type":"ContainerStarted","Data":"f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2"}
Jul 28 11:09:18.068673 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:18.068630    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:18.070017 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:18.069992    2578 patch_prober.go:28] interesting pod/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 11:09:18.070132 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:18.070065    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 11:09:19.069090 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:19.069046    2578 patch_prober.go:28] interesting pod/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 11:09:19.069523 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:19.069107    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 11:09:20.068938 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:20.068888    2578 patch_prober.go:28] interesting pod/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 11:09:20.069253 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:20.068958    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 11:09:21.068981 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:21.068933    2578 patch_prober.go:28] interesting pod/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Jul 28 11:09:21.069244 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:21.069014    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Jul 28 11:09:22.073060 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:22.073020    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:22.073541 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:22.073311    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:22.074071 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:22.074054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:25.331581 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.331542    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"]
Jul 28 11:09:25.335161 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.335142    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:25.337500 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.337475    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:09:25.337642 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.337521    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 11:09:25.342772 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.342453    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"]
Jul 28 11:09:25.466581 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.466547    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bc7d66-6a5b-42d0-a863-6f790407b9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1912e6-9wghz\" (UID: \"e0bc7d66-6a5b-42d0-a863-6f790407b9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:25.567194 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.567156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bc7d66-6a5b-42d0-a863-6f790407b9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1912e6-9wghz\" (UID: \"e0bc7d66-6a5b-42d0-a863-6f790407b9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:25.567826 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.567805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bc7d66-6a5b-42d0-a863-6f790407b9d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1912e6-9wghz\" (UID: \"e0bc7d66-6a5b-42d0-a863-6f790407b9d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:25.646764 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.646687    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:25.960838 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.960803    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:09:25.961021 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.960994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:09:25.990481 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:25.990454    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"]
Jul 28 11:09:25.993077 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:09:25.993043    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bc7d66_6a5b_42d0_a863_6f790407b9d2.slice/crio-61ab4084a37829bb106fa5b123511cf217f2b145cbc22f36fae7ca35b6fbbb41 WatchSource:0}: Error finding container 61ab4084a37829bb106fa5b123511cf217f2b145cbc22f36fae7ca35b6fbbb41: Status 404 returned error can't find the container with id 61ab4084a37829bb106fa5b123511cf217f2b145cbc22f36fae7ca35b6fbbb41
Jul 28 11:09:26.017639 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:26.017608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz" event={"ID":"e0bc7d66-6a5b-42d0-a863-6f790407b9d2","Type":"ContainerStarted","Data":"61ab4084a37829bb106fa5b123511cf217f2b145cbc22f36fae7ca35b6fbbb41"}
Jul 28 11:09:27.024015 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:27.023921    2578 generic.go:356] "Generic (PLEG): container finished" podID="e0bc7d66-6a5b-42d0-a863-6f790407b9d2" containerID="9a056a97e51c2c8d9f2f4ddc63e892703013a20e2c4da0479ecc19adeae493c6" exitCode=0
Jul 28 11:09:27.024015 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:27.024003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz" event={"ID":"e0bc7d66-6a5b-42d0-a863-6f790407b9d2","Type":"ContainerDied","Data":"9a056a97e51c2c8d9f2f4ddc63e892703013a20e2c4da0479ecc19adeae493c6"}
Jul 28 11:09:28.161840 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:28.161809    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:28.291565 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:28.291470    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bc7d66-6a5b-42d0-a863-6f790407b9d2-maas-api-service-ca\") pod \"e0bc7d66-6a5b-42d0-a863-6f790407b9d2\" (UID: \"e0bc7d66-6a5b-42d0-a863-6f790407b9d2\") "
Jul 28 11:09:28.291867 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:28.291844    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bc7d66-6a5b-42d0-a863-6f790407b9d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0bc7d66-6a5b-42d0-a863-6f790407b9d2" (UID: "e0bc7d66-6a5b-42d0-a863-6f790407b9d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:09:28.392551 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:28.392514    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bc7d66-6a5b-42d0-a863-6f790407b9d2-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:29.034204 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:29.034175    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"
Jul 28 11:09:29.034204 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:29.034197    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz" event={"ID":"e0bc7d66-6a5b-42d0-a863-6f790407b9d2","Type":"ContainerDied","Data":"61ab4084a37829bb106fa5b123511cf217f2b145cbc22f36fae7ca35b6fbbb41"}
Jul 28 11:09:29.034447 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:29.034246    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61ab4084a37829bb106fa5b123511cf217f2b145cbc22f36fae7ca35b6fbbb41"
Jul 28 11:09:30.143205 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:30.143172    2578 scope.go:117] "RemoveContainer" containerID="ed3642c91ce2537a5dcd12001810cf2e2fa8272b783d469c5769a048e2864f32"
Jul 28 11:09:41.041613 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.041571    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"]
Jul 28 11:09:41.042084 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.041791    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz" podUID="da6cc156-f924-4f66-8460-c522fd30fc59" containerName="payload-processing" containerID="cri-o://47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c" gracePeriod=30
Jul 28 11:09:41.294544 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.294470    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:41.404934 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.404895    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6cc156-f924-4f66-8460-c522fd30fc59-plugins-config-volume\") pod \"da6cc156-f924-4f66-8460-c522fd30fc59\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") "
Jul 28 11:09:41.405106 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.405013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dzdb\" (UniqueName: \"kubernetes.io/projected/da6cc156-f924-4f66-8460-c522fd30fc59-kube-api-access-9dzdb\") pod \"da6cc156-f924-4f66-8460-c522fd30fc59\" (UID: \"da6cc156-f924-4f66-8460-c522fd30fc59\") "
Jul 28 11:09:41.405350 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.405318    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6cc156-f924-4f66-8460-c522fd30fc59-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "da6cc156-f924-4f66-8460-c522fd30fc59" (UID: "da6cc156-f924-4f66-8460-c522fd30fc59"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:09:41.407224 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.407198    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6cc156-f924-4f66-8460-c522fd30fc59-kube-api-access-9dzdb" (OuterVolumeSpecName: "kube-api-access-9dzdb") pod "da6cc156-f924-4f66-8460-c522fd30fc59" (UID: "da6cc156-f924-4f66-8460-c522fd30fc59"). InnerVolumeSpecName "kube-api-access-9dzdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:09:41.505760 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.505732    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dzdb\" (UniqueName: \"kubernetes.io/projected/da6cc156-f924-4f66-8460-c522fd30fc59-kube-api-access-9dzdb\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:41.505760 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:41.505759    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da6cc156-f924-4f66-8460-c522fd30fc59-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:42.098608 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.098564    2578 generic.go:356] "Generic (PLEG): container finished" podID="da6cc156-f924-4f66-8460-c522fd30fc59" containerID="47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c" exitCode=0
Jul 28 11:09:42.099061 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.098632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"
Jul 28 11:09:42.099061 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.098649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz" event={"ID":"da6cc156-f924-4f66-8460-c522fd30fc59","Type":"ContainerDied","Data":"47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c"}
Jul 28 11:09:42.099061 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.098729    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz" event={"ID":"da6cc156-f924-4f66-8460-c522fd30fc59","Type":"ContainerDied","Data":"417ffe92c817b0bb652d539e5a393627fe0bc2296a84f3d0c8d1a18a156a5dbc"}
Jul 28 11:09:42.099061 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.098752    2578 scope.go:117] "RemoveContainer" containerID="47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c"
Jul 28 11:09:42.107307 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.107289    2578 scope.go:117] "RemoveContainer" containerID="47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c"
Jul 28 11:09:42.107624 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:09:42.107606    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c\": container with ID starting with 47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c not found: ID does not exist" containerID="47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c"
Jul 28 11:09:42.107690 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.107636    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c"} err="failed to get container status \"47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c\": rpc error: code = NotFound desc = could not find container \"47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c\": container with ID starting with 47278dd65c024687698c323b7571185b583654f301fdfef3e68c06dfd1dd710c not found: ID does not exist"
Jul 28 11:09:42.114398 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.114369    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"]
Jul 28 11:09:42.117918 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:42.117890    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-1912e6-75d95f845b-sjlsz"]
Jul 28 11:09:43.494669 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:43.494633    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6cc156-f924-4f66-8460-c522fd30fc59" path="/var/lib/kubelet/pods/da6cc156-f924-4f66-8460-c522fd30fc59/volumes"
Jul 28 11:09:45.401648 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:45.401599    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"]
Jul 28 11:09:45.403342 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:45.403316    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1912e6-9wghz"]
Jul 28 11:09:45.494441 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:45.494404    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bc7d66-6a5b-42d0-a863-6f790407b9d2" path="/var/lib/kubelet/pods/e0bc7d66-6a5b-42d0-a863-6f790407b9d2/volumes"
Jul 28 11:09:51.755128 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:51.755036    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"]
Jul 28 11:09:51.755702 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:51.755411    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy" containerID="cri-o://f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2" gracePeriod=30
Jul 28 11:09:52.076183 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:52.076085    2578 patch_prober.go:28] interesting pod/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 11:09:52.076183 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:52.076147    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 11:09:57.017779 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.017750    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:57.153571 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153455    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-podinfo\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.153571 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153526    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-credential-socket\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.153571 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153573    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-data\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.153833 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153674    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istiod-ca-cert\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.153833 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153708    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-envoy\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.153833 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153739    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p5dd\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-kube-api-access-6p5dd\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.153833 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153814    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-token\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.154051 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-certs\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.154051 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153860    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:09:57.154051 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.153916    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-socket\") pod \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\" (UID: \"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a\") "
Jul 28 11:09:57.154227 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.154053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:09:57.154227 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.154204    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.154382 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.154229    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.154616 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.154500    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:09:57.154698 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.154672    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:09:57.154847 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.154821    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:09:57.155998 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.155971    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:09:57.156118 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.156005    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-kube-api-access-6p5dd" (OuterVolumeSpecName: "kube-api-access-6p5dd") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "kube-api-access-6p5dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:09:57.156118 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.156023    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:09:57.156202 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.156127    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" (UID: "7f21d2e6-f74d-4071-9a02-5969cb8e4f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:09:57.173363 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.173330    2578 generic.go:356] "Generic (PLEG): container finished" podID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerID="f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2" exitCode=0
Jul 28 11:09:57.173529 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.173381    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"
Jul 28 11:09:57.173529 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.173393    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" event={"ID":"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a","Type":"ContainerDied","Data":"f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2"}
Jul 28 11:09:57.173529 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.173438    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc" event={"ID":"7f21d2e6-f74d-4071-9a02-5969cb8e4f0a","Type":"ContainerDied","Data":"234c4e8c81cac425d105b7679d785d731723f123cd0bc44de16b3f5ce4aa8abe"}
Jul 28 11:09:57.173529 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.173460    2578 scope.go:117] "RemoveContainer" containerID="f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2"
Jul 28 11:09:57.183133 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.183109    2578 scope.go:117] "RemoveContainer" containerID="0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"
Jul 28 11:09:57.191778 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.191753    2578 scope.go:117] "RemoveContainer" containerID="f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2"
Jul 28 11:09:57.192093 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:09:57.192073    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2\": container with ID starting with f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2 not found: ID does not exist" containerID="f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2"
Jul 28 11:09:57.192142 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.192105    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2"} err="failed to get container status \"f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2\": rpc error: code = NotFound desc = could not find container \"f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2\": container with ID starting with f87dac325f8f991bd7c27d467f1b6cb72e23777b97c3a7a46f1c2c65752369e2 not found: ID does not exist"
Jul 28 11:09:57.192142 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.192127    2578 scope.go:117] "RemoveContainer" containerID="0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"
Jul 28 11:09:57.192434 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:09:57.192411    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e\": container with ID starting with 0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e not found: ID does not exist" containerID="0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"
Jul 28 11:09:57.192487 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.192441    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e"} err="failed to get container status \"0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e\": rpc error: code = NotFound desc = could not find container \"0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e\": container with ID starting with 0fc06467cdcacb376df05377bf580de0791b70d24afac8ba7c6ca374c799c80e not found: ID does not exist"
Jul 28 11:09:57.196075 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.196045    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"]
Jul 28 11:09:57.199493 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.199465    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1912e6-openshift-default-847bb57d7b-28rxc"]
Jul 28 11:09:57.254970 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.254931    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.254970 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.254964    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.254970 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.254974    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.255217 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.254984    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.255217 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.254992    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5dd\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-kube-api-access-6p5dd\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.255217 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.255002    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.255217 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.255011    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:09:57.494225 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:09:57.494184    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" path="/var/lib/kubelet/pods/7f21d2e6-f74d-4071-9a02-5969cb8e4f0a/volumes"
Jul 28 11:10:07.590405 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.590368    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"]
Jul 28 11:10:07.590931 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.590583    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b" podUID="16e95252-5de3-4dcf-a48a-e752766529d4" containerName="payload-processing" containerID="cri-o://a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999" gracePeriod=30
Jul 28 11:10:07.598616 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.598571    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"]
Jul 28 11:10:07.599125 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.598844    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl" podUID="cf65c864-692c-4605-8eb6-f539ac6ac400" containerName="payload-pre-processing" containerID="cri-o://cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e" gracePeriod=30
Jul 28 11:10:07.868193 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.868152    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:10:07.871158 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.871131    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:10:07.945955 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.945921    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8b85\" (UniqueName: \"kubernetes.io/projected/16e95252-5de3-4dcf-a48a-e752766529d4-kube-api-access-t8b85\") pod \"16e95252-5de3-4dcf-a48a-e752766529d4\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") "
Jul 28 11:10:07.946114 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.945985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e95252-5de3-4dcf-a48a-e752766529d4-plugins-config-volume\") pod \"16e95252-5de3-4dcf-a48a-e752766529d4\" (UID: \"16e95252-5de3-4dcf-a48a-e752766529d4\") "
Jul 28 11:10:07.946114 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.946005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65c864-692c-4605-8eb6-f539ac6ac400-plugins-config-volume\") pod \"cf65c864-692c-4605-8eb6-f539ac6ac400\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") "
Jul 28 11:10:07.946114 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.946042    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jqq7\" (UniqueName: \"kubernetes.io/projected/cf65c864-692c-4605-8eb6-f539ac6ac400-kube-api-access-5jqq7\") pod \"cf65c864-692c-4605-8eb6-f539ac6ac400\" (UID: \"cf65c864-692c-4605-8eb6-f539ac6ac400\") "
Jul 28 11:10:07.946435 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.946410    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf65c864-692c-4605-8eb6-f539ac6ac400-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf65c864-692c-4605-8eb6-f539ac6ac400" (UID: "cf65c864-692c-4605-8eb6-f539ac6ac400"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:10:07.946514 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.946436    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e95252-5de3-4dcf-a48a-e752766529d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16e95252-5de3-4dcf-a48a-e752766529d4" (UID: "16e95252-5de3-4dcf-a48a-e752766529d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:10:07.948445 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.948415    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf65c864-692c-4605-8eb6-f539ac6ac400-kube-api-access-5jqq7" (OuterVolumeSpecName: "kube-api-access-5jqq7") pod "cf65c864-692c-4605-8eb6-f539ac6ac400" (UID: "cf65c864-692c-4605-8eb6-f539ac6ac400"). InnerVolumeSpecName "kube-api-access-5jqq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:10:07.948534 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:07.948479    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e95252-5de3-4dcf-a48a-e752766529d4-kube-api-access-t8b85" (OuterVolumeSpecName: "kube-api-access-t8b85") pod "16e95252-5de3-4dcf-a48a-e752766529d4" (UID: "16e95252-5de3-4dcf-a48a-e752766529d4"). InnerVolumeSpecName "kube-api-access-t8b85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:10:08.047172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.047134    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16e95252-5de3-4dcf-a48a-e752766529d4-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:10:08.047172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.047167    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf65c864-692c-4605-8eb6-f539ac6ac400-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:10:08.047172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.047177    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jqq7\" (UniqueName: \"kubernetes.io/projected/cf65c864-692c-4605-8eb6-f539ac6ac400-kube-api-access-5jqq7\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:10:08.047172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.047185    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8b85\" (UniqueName: \"kubernetes.io/projected/16e95252-5de3-4dcf-a48a-e752766529d4-kube-api-access-t8b85\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:10:08.227037 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.226999    2578 generic.go:356] "Generic (PLEG): container finished" podID="cf65c864-692c-4605-8eb6-f539ac6ac400" containerID="cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e" exitCode=0
Jul 28 11:10:08.227255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.227049    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"
Jul 28 11:10:08.227255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.227077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl" event={"ID":"cf65c864-692c-4605-8eb6-f539ac6ac400","Type":"ContainerDied","Data":"cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e"}
Jul 28 11:10:08.227255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.227140    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl" event={"ID":"cf65c864-692c-4605-8eb6-f539ac6ac400","Type":"ContainerDied","Data":"16fc6ecee7882ac18a0985a22429ddd1b84d3f35a2784e3f137b1239ba86d80a"}
Jul 28 11:10:08.227255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.227176    2578 scope.go:117] "RemoveContainer" containerID="cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e"
Jul 28 11:10:08.228527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.228450    2578 generic.go:356] "Generic (PLEG): container finished" podID="16e95252-5de3-4dcf-a48a-e752766529d4" containerID="a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999" exitCode=0
Jul 28 11:10:08.228527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.228505    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"
Jul 28 11:10:08.228527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.228520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b" event={"ID":"16e95252-5de3-4dcf-a48a-e752766529d4","Type":"ContainerDied","Data":"a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999"}
Jul 28 11:10:08.228752 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.228563    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b" event={"ID":"16e95252-5de3-4dcf-a48a-e752766529d4","Type":"ContainerDied","Data":"e31c291e977d5c1beee43fdbe062111082417bc865ea1134b27fb83fda2929ee"}
Jul 28 11:10:08.237533 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.237359    2578 scope.go:117] "RemoveContainer" containerID="cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e"
Jul 28 11:10:08.237863 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:10:08.237843    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e\": container with ID starting with cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e not found: ID does not exist" containerID="cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e"
Jul 28 11:10:08.237940 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.237873    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e"} err="failed to get container status \"cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e\": rpc error: code = NotFound desc = could not find container \"cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e\": container with ID starting with cc4d92cb63c233711241f6b7a8feaad654f74333281bdb678c36febee9ef3b4e not found: ID does not exist"
Jul 28 11:10:08.237940 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.237892    2578 scope.go:117] "RemoveContainer" containerID="a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999"
Jul 28 11:10:08.246682 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.246658    2578 scope.go:117] "RemoveContainer" containerID="a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999"
Jul 28 11:10:08.246982 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:10:08.246961    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999\": container with ID starting with a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999 not found: ID does not exist" containerID="a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999"
Jul 28 11:10:08.247055 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.246991    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999"} err="failed to get container status \"a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999\": rpc error: code = NotFound desc = could not find container \"a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999\": container with ID starting with a381f114c0ce9b47b25646c7e9d41f8e14b98333db4748001ec144d84d712999 not found: ID does not exist"
Jul 28 11:10:08.252884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.252849    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"]
Jul 28 11:10:08.257079 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.257048    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-86170f-6df496f444-9c4xl"]
Jul 28 11:10:08.266778 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.266747    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"]
Jul 28 11:10:08.272650 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:08.272621    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-86170f-6df5c6bf6b-fkc8b"]
Jul 28 11:10:09.496713 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:09.496675    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e95252-5de3-4dcf-a48a-e752766529d4" path="/var/lib/kubelet/pods/16e95252-5de3-4dcf-a48a-e752766529d4/volumes"
Jul 28 11:10:09.497099 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:09.497024    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf65c864-692c-4605-8eb6-f539ac6ac400" path="/var/lib/kubelet/pods/cf65c864-692c-4605-8eb6-f539ac6ac400/volumes"
Jul 28 11:10:19.654781 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.654743    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"]
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655061    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e95252-5de3-4dcf-a48a-e752766529d4" containerName="payload-processing"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655075    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e95252-5de3-4dcf-a48a-e752766529d4" containerName="payload-processing"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655101    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6cc156-f924-4f66-8460-c522fd30fc59" containerName="payload-processing"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655110    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6cc156-f924-4f66-8460-c522fd30fc59" containerName="payload-processing"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655131    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf65c864-692c-4605-8eb6-f539ac6ac400" containerName="payload-pre-processing"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655143    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf65c864-692c-4605-8eb6-f539ac6ac400" containerName="payload-pre-processing"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655157    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655168    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655182    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655190    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655206    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bc7d66-6a5b-42d0-a863-6f790407b9d2" containerName="revoke-keys"
Jul 28 11:10:19.655303 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655213    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bc7d66-6a5b-42d0-a863-6f790407b9d2" containerName="revoke-keys"
Jul 28 11:10:19.655745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655320    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy"
Jul 28 11:10:19.655745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655335    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f21d2e6-f74d-4071-9a02-5969cb8e4f0a" containerName="istio-proxy"
Jul 28 11:10:19.655745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655345    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bc7d66-6a5b-42d0-a863-6f790407b9d2" containerName="revoke-keys"
Jul 28 11:10:19.655745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655352    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf65c864-692c-4605-8eb6-f539ac6ac400" containerName="payload-pre-processing"
Jul 28 11:10:19.655745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655360    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e95252-5de3-4dcf-a48a-e752766529d4" containerName="payload-processing"
Jul 28 11:10:19.655745 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.655370    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6cc156-f924-4f66-8460-c522fd30fc59" containerName="payload-processing"
Jul 28 11:10:19.660112 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.660085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.662705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.662675    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 11:10:19.662836 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.662679    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-4cda06-openshift-default-dockercfg-2b6xd\""
Jul 28 11:10:19.672343 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.672314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"]
Jul 28 11:10:19.853400 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853365    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-credential-socket\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853400 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853404    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istiod-ca-cert\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853425    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-data\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853448    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-certs\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-token\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-socket\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-envoy\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853834 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853648    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kz2m\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-kube-api-access-2kz2m\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.853834 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.853693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-podinfo\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kz2m\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-kube-api-access-2kz2m\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-podinfo\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-credential-socket\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955178    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istiod-ca-cert\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-data\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-certs\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-token\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955335    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-socket\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955372    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-envoy\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-credential-socket\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.955756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.955733    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-socket\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.956131 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.956106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-certs\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.956131 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.956107    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-data\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.956351 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.956146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istiod-ca-cert\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.957811 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.957792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-envoy\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.957928 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.957909    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-podinfo\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.963622 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.963590    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kz2m\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-kube-api-access-2kz2m\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.963733 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.963600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-token\") pod \"e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") " pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:19.970625 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:19.970596    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:20.114420 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.114212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"]
Jul 28 11:10:20.117765 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:10:20.117727    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb130dafc_5b3d_4a62_bc3d_27087f3d3535.slice/crio-2a7d05ecbee695f8046180c8692e3f12164fac4d5e36a917e13019f0b915122b WatchSource:0}: Error finding container 2a7d05ecbee695f8046180c8692e3f12164fac4d5e36a917e13019f0b915122b: Status 404 returned error can't find the container with id 2a7d05ecbee695f8046180c8692e3f12164fac4d5e36a917e13019f0b915122b
Jul 28 11:10:20.120022 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.119979    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:10:20.120121 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.120075    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:10:20.120175 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.120121    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:10:20.286757 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.286655    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" event={"ID":"b130dafc-5b3d-4a62-bc3d-27087f3d3535","Type":"ContainerStarted","Data":"d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"}
Jul 28 11:10:20.286757 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.286715    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" event={"ID":"b130dafc-5b3d-4a62-bc3d-27087f3d3535","Type":"ContainerStarted","Data":"2a7d05ecbee695f8046180c8692e3f12164fac4d5e36a917e13019f0b915122b"}
Jul 28 11:10:20.306008 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.305935    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podStartSLOduration=1.305911265 podStartE2EDuration="1.305911265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:20.304799144 +0000 UTC m=+2811.346891309" watchObservedRunningTime="2026-07-28 11:10:20.305911265 +0000 UTC m=+2811.348003420"
Jul 28 11:10:20.971376 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:20.971339    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:21.976441 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:21.976411    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:22.297228 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:22.297145    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:22.298222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:22.298203    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:23.125025 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.124987    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"]
Jul 28 11:10:23.128495 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.128473    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.131024 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.130991    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-4cda06\""
Jul 28 11:10:23.131189 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.131042    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-4cda06-dockercfg-8w4vr\""
Jul 28 11:10:23.139327 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.139291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"]
Jul 28 11:10:23.181448 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.181408    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293f698-7836-4336-bcf1-b0a1fde79ebf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.181606 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.181479    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94wnq\" (UniqueName: \"kubernetes.io/projected/c293f698-7836-4336-bcf1-b0a1fde79ebf-kube-api-access-94wnq\") pod \"payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.281979 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.281932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293f698-7836-4336-bcf1-b0a1fde79ebf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.282206 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.282019    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94wnq\" (UniqueName: \"kubernetes.io/projected/c293f698-7836-4336-bcf1-b0a1fde79ebf-kube-api-access-94wnq\") pod \"payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.282744 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.282719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293f698-7836-4336-bcf1-b0a1fde79ebf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.290615 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.290577    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-94wnq\" (UniqueName: \"kubernetes.io/projected/c293f698-7836-4336-bcf1-b0a1fde79ebf-kube-api-access-94wnq\") pod \"payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.440494 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.440453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:23.575029 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:23.574984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"]
Jul 28 11:10:23.576962 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:10:23.576934    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc293f698_7836_4336_bcf1_b0a1fde79ebf.slice/crio-18fe6b6b4c0006cf046b63dc500c23e68d160f59f2bc83dec76402c43bd06694 WatchSource:0}: Error finding container 18fe6b6b4c0006cf046b63dc500c23e68d160f59f2bc83dec76402c43bd06694: Status 404 returned error can't find the container with id 18fe6b6b4c0006cf046b63dc500c23e68d160f59f2bc83dec76402c43bd06694
Jul 28 11:10:24.309293 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:24.309219    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r" event={"ID":"c293f698-7836-4336-bcf1-b0a1fde79ebf","Type":"ContainerStarted","Data":"e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6"}
Jul 28 11:10:24.309780 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:24.309311    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r" event={"ID":"c293f698-7836-4336-bcf1-b0a1fde79ebf","Type":"ContainerStarted","Data":"18fe6b6b4c0006cf046b63dc500c23e68d160f59f2bc83dec76402c43bd06694"}
Jul 28 11:10:24.309780 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:24.309453    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:24.328013 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:24.327951    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r" podStartSLOduration=1.3279313990000001 podStartE2EDuration="1.327931399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:24.327588909 +0000 UTC m=+2815.369681077" watchObservedRunningTime="2026-07-28 11:10:24.327931399 +0000 UTC m=+2815.370023552"
Jul 28 11:10:30.977144 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:30.977104    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"]
Jul 28 11:10:30.981985 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:30.981962    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:30.984893 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:30.984868    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-683bdc\""
Jul 28 11:10:30.995348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:30.995318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"]
Jul 28 11:10:31.032188 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.032149    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"]
Jul 28 11:10:31.035514 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.035486    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.038161 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.038141    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-683bdc-dockercfg-glp4b\""
Jul 28 11:10:31.045841 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.045819    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwhw\" (UniqueName: \"kubernetes.io/projected/b35a494e-db46-4d09-a16b-b2883577e13b-kube-api-access-nwwhw\") pod \"payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.045992 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.045857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2zkv\" (UniqueName: \"kubernetes.io/projected/03a7bd96-1a17-4490-8ea8-b0697ac08239-kube-api-access-q2zkv\") pod \"payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.045992 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.045960    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35a494e-db46-4d09-a16b-b2883577e13b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.046078 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.046013    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a7bd96-1a17-4490-8ea8-b0697ac08239-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.050233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.050204    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"]
Jul 28 11:10:31.147125 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.147083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a7bd96-1a17-4490-8ea8-b0697ac08239-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.147363 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.147133    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwhw\" (UniqueName: \"kubernetes.io/projected/b35a494e-db46-4d09-a16b-b2883577e13b-kube-api-access-nwwhw\") pod \"payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.147363 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.147257    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2zkv\" (UniqueName: \"kubernetes.io/projected/03a7bd96-1a17-4490-8ea8-b0697ac08239-kube-api-access-q2zkv\") pod \"payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.147496 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.147374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35a494e-db46-4d09-a16b-b2883577e13b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.153543 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.148465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a7bd96-1a17-4490-8ea8-b0697ac08239-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.153543 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.148694    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35a494e-db46-4d09-a16b-b2883577e13b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.157189 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.157155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwhw\" (UniqueName: \"kubernetes.io/projected/b35a494e-db46-4d09-a16b-b2883577e13b-kube-api-access-nwwhw\") pod \"payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.157363 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.157255    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2zkv\" (UniqueName: \"kubernetes.io/projected/03a7bd96-1a17-4490-8ea8-b0697ac08239-kube-api-access-q2zkv\") pod \"payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.293472 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.293368    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:31.345812 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.345781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:31.444819 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.444797    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"]
Jul 28 11:10:31.447873 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:10:31.447836    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35a494e_db46_4d09_a16b_b2883577e13b.slice/crio-ffa0f8463f3091e371797ce3314fa8edee99579b66ce714f4a97faf5e318a453 WatchSource:0}: Error finding container ffa0f8463f3091e371797ce3314fa8edee99579b66ce714f4a97faf5e318a453: Status 404 returned error can't find the container with id ffa0f8463f3091e371797ce3314fa8edee99579b66ce714f4a97faf5e318a453
Jul 28 11:10:31.503740 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:31.503696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"]
Jul 28 11:10:31.507803 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:10:31.507777    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a7bd96_1a17_4490_8ea8_b0697ac08239.slice/crio-ff534c055837a8b33bbc1eb12a67012397388f3c068de9a0216a59023985c631 WatchSource:0}: Error finding container ff534c055837a8b33bbc1eb12a67012397388f3c068de9a0216a59023985c631: Status 404 returned error can't find the container with id ff534c055837a8b33bbc1eb12a67012397388f3c068de9a0216a59023985c631
Jul 28 11:10:32.353441 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.353383    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph" event={"ID":"03a7bd96-1a17-4490-8ea8-b0697ac08239","Type":"ContainerStarted","Data":"6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444"}
Jul 28 11:10:32.353901 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.353473    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:32.353901 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.353497    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph" event={"ID":"03a7bd96-1a17-4490-8ea8-b0697ac08239","Type":"ContainerStarted","Data":"ff534c055837a8b33bbc1eb12a67012397388f3c068de9a0216a59023985c631"}
Jul 28 11:10:32.355005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.354977    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6" event={"ID":"b35a494e-db46-4d09-a16b-b2883577e13b","Type":"ContainerStarted","Data":"b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c"}
Jul 28 11:10:32.355126 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.355023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6" event={"ID":"b35a494e-db46-4d09-a16b-b2883577e13b","Type":"ContainerStarted","Data":"ffa0f8463f3091e371797ce3314fa8edee99579b66ce714f4a97faf5e318a453"}
Jul 28 11:10:32.355126 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.355100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:32.370435 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.370385    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph" podStartSLOduration=1.37036703 podStartE2EDuration="1.37036703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:32.36773916 +0000 UTC m=+2823.409831314" watchObservedRunningTime="2026-07-28 11:10:32.37036703 +0000 UTC m=+2823.412459182"
Jul 28 11:10:32.384043 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:32.383985    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6" podStartSLOduration=2.383970227 podStartE2EDuration="2.383970227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:32.38256607 +0000 UTC m=+2823.424658413" watchObservedRunningTime="2026-07-28 11:10:32.383970227 +0000 UTC m=+2823.426062379"
Jul 28 11:10:35.321502 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:35.321467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:10:38.338496 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:38.338449    2578 patch_prober.go:28] interesting pod/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 11:10:38.338898 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:38.338529    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 11:10:39.391381 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:39.391344    2578 generic.go:356] "Generic (PLEG): container finished" podID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerID="d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08" exitCode=0
Jul 28 11:10:39.391937 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:39.391425    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" event={"ID":"b130dafc-5b3d-4a62-bc3d-27087f3d3535","Type":"ContainerDied","Data":"d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"}
Jul 28 11:10:39.392017 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:39.391940    2578 scope.go:117] "RemoveContainer" containerID="d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"
Jul 28 11:10:39.393038 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:39.393006    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:10:39.393141 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:39.393092    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:10:39.393206 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:39.393139    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:10:40.398353 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:40.398314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" event={"ID":"b130dafc-5b3d-4a62-bc3d-27087f3d3535","Type":"ContainerStarted","Data":"77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad"}
Jul 28 11:10:40.971452 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:40.971409    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:40.971703 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:40.971678    2578 patch_prober.go:28] interesting pod/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 28 11:10:40.971761 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:40.971741    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 28 11:10:41.971915 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:41.971872    2578 patch_prober.go:28] interesting pod/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 28 11:10:41.972432 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:41.971946    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 28 11:10:42.971955 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:42.971915    2578 patch_prober.go:28] interesting pod/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 28 11:10:42.972406 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:42.971979    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 28 11:10:43.362413 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:43.362379    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:10:43.362584 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:43.362558    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:10:43.972005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:43.971965    2578 patch_prober.go:28] interesting pod/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Jul 28 11:10:43.972443 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:43.972026    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Jul 28 11:10:44.974693 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:44.974661    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:44.975068 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:44.974994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:44.975727 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:44.975710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:10:50.948084 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.948047    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"]
Jul 28 11:10:50.952187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.952156    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:50.954578 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.954551    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"openshift-service-ca.crt\""
Jul 28 11:10:50.955557 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.955529    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"default-dockercfg-rffrz\""
Jul 28 11:10:50.955557 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.955549    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"auth-test-model-4cda06-kserve-self-signed-certs\""
Jul 28 11:10:50.955740 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.955541    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"kube-root-ca.crt\""
Jul 28 11:10:50.961329 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:50.960008    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"]
Jul 28 11:10:51.022421 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.022380    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-home\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.022638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.022432    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77p5n\" (UniqueName: \"kubernetes.io/projected/c9345a0f-7747-49b7-8fd6-b0aab32f518d-kube-api-access-77p5n\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.022638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.022499    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-model-cache\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.022638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.022539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tmp-dir\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.022638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.022574    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tls-certs\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.022638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.022626    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-dshm\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124016 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.123981    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-home\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124207 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77p5n\" (UniqueName: \"kubernetes.io/projected/c9345a0f-7747-49b7-8fd6-b0aab32f518d-kube-api-access-77p5n\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124207 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-model-cache\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124207 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tmp-dir\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124207 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tls-certs\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124207 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124168    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-dshm\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124532 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-model-cache\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124532 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-home\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.124655 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.124574    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tmp-dir\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.126720 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.126657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-dshm\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.127045 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.127025    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tls-certs\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.132119 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.132092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77p5n\" (UniqueName: \"kubernetes.io/projected/c9345a0f-7747-49b7-8fd6-b0aab32f518d-kube-api-access-77p5n\") pod \"auth-test-model-4cda06-kserve-6847674875-stww4\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") " pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.264239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.264134    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:10:51.405015 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.404987    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"]
Jul 28 11:10:51.407506 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:10:51.407472    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9345a0f_7747_49b7_8fd6_b0aab32f518d.slice/crio-c07cd87687a8b89ddbc68bc84093573c5c5ac80b1543c3b8f67f4e5a76a52ae4 WatchSource:0}: Error finding container c07cd87687a8b89ddbc68bc84093573c5c5ac80b1543c3b8f67f4e5a76a52ae4: Status 404 returned error can't find the container with id c07cd87687a8b89ddbc68bc84093573c5c5ac80b1543c3b8f67f4e5a76a52ae4
Jul 28 11:10:51.478622 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.478575    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4" event={"ID":"c9345a0f-7747-49b7-8fd6-b0aab32f518d","Type":"ContainerStarted","Data":"c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db"}
Jul 28 11:10:51.478804 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.478635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4" event={"ID":"c9345a0f-7747-49b7-8fd6-b0aab32f518d","Type":"ContainerStarted","Data":"c07cd87687a8b89ddbc68bc84093573c5c5ac80b1543c3b8f67f4e5a76a52ae4"}
Jul 28 11:10:51.496786 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:10:51.496735    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4" podStartSLOduration=1.496716417 podStartE2EDuration="1.496716417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:10:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:10:51.494300071 +0000 UTC m=+2842.536392225" watchObservedRunningTime="2026-07-28 11:10:51.496716417 +0000 UTC m=+2842.538808569"
Jul 28 11:11:01.264683 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:11:01.264641    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:11:01.265117 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:11:01.264708    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:11:01.277572 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:11:01.277544    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:11:01.537001 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:11:01.536909    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:13:30.030891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:13:30.030783    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:13:30.041937 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:13:30.041911    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:15:00.138054 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.138009    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"]
Jul 28 11:15:00.142368 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.142342    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"]
Jul 28 11:15:00.142488 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.142468    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"
Jul 28 11:15:00.144905 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.144882    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:15:00.146026 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.146000    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"
Jul 28 11:15:00.154088 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.154060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"
Jul 28 11:15:00.157096 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.157064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"]
Jul 28 11:15:00.159519 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.159488    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"]
Jul 28 11:15:00.164385 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.164362    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"
Jul 28 11:15:00.315322 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.315282    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"]
Jul 28 11:15:00.315812 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:15:00.315782    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25fdb20_b2a6_489a_9dd3_7aaf067c9508.slice/crio-5d96dde903e5155d0af8c5a5751a7de9522b5dbabccf327cd7ba3b4dabf7ed48 WatchSource:0}: Error finding container 5d96dde903e5155d0af8c5a5751a7de9522b5dbabccf327cd7ba3b4dabf7ed48: Status 404 returned error can't find the container with id 5d96dde903e5155d0af8c5a5751a7de9522b5dbabccf327cd7ba3b4dabf7ed48
Jul 28 11:15:00.341396 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.341365    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"]
Jul 28 11:15:00.351584 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:15:00.351554    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b29d3b2_0957_46a6_a304_c085cbf41fb2.slice/crio-12d06fdde5a9dbdeda99b734577d777d753a5e7f01c8eac332bffed94d6825fa WatchSource:0}: Error finding container 12d06fdde5a9dbdeda99b734577d777d753a5e7f01c8eac332bffed94d6825fa: Status 404 returned error can't find the container with id 12d06fdde5a9dbdeda99b734577d777d753a5e7f01c8eac332bffed94d6825fa
Jul 28 11:15:00.665407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.665259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v" event={"ID":"b25fdb20-b2a6-489a-9dd3-7aaf067c9508","Type":"ContainerStarted","Data":"744681982f4f03e87fc4565697c1ca071301c423d11dd51dab551531a8b3b709"}
Jul 28 11:15:00.665407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.665354    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v" event={"ID":"b25fdb20-b2a6-489a-9dd3-7aaf067c9508","Type":"ContainerStarted","Data":"5d96dde903e5155d0af8c5a5751a7de9522b5dbabccf327cd7ba3b4dabf7ed48"}
Jul 28 11:15:00.666905 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.666879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz" event={"ID":"8b29d3b2-0957-46a6-a304-c085cbf41fb2","Type":"ContainerStarted","Data":"ca57b1d0d4dab17a3c09f6eb7423a49dba5582ad45ce2f2e76ee165762892f32"}
Jul 28 11:15:00.667030 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.666927    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz" event={"ID":"8b29d3b2-0957-46a6-a304-c085cbf41fb2","Type":"ContainerStarted","Data":"12d06fdde5a9dbdeda99b734577d777d753a5e7f01c8eac332bffed94d6825fa"}
Jul 28 11:15:00.680048 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.679991    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v" podStartSLOduration=0.679977538 podStartE2EDuration="679.977538ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:15:00.679216444 +0000 UTC m=+3091.721308666" watchObservedRunningTime="2026-07-28 11:15:00.679977538 +0000 UTC m=+3091.722069691"
Jul 28 11:15:00.696885 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:00.696818    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz" podStartSLOduration=0.696795559 podStartE2EDuration="696.795559ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:15:00.692812873 +0000 UTC m=+3091.734905027" watchObservedRunningTime="2026-07-28 11:15:00.696795559 +0000 UTC m=+3091.738887715"
Jul 28 11:15:01.674255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:01.674217    2578 generic.go:356] "Generic (PLEG): container finished" podID="b25fdb20-b2a6-489a-9dd3-7aaf067c9508" containerID="744681982f4f03e87fc4565697c1ca071301c423d11dd51dab551531a8b3b709" exitCode=0
Jul 28 11:15:01.674758 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:01.674324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v" event={"ID":"b25fdb20-b2a6-489a-9dd3-7aaf067c9508","Type":"ContainerDied","Data":"744681982f4f03e87fc4565697c1ca071301c423d11dd51dab551531a8b3b709"}
Jul 28 11:15:01.675990 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:01.675966    2578 generic.go:356] "Generic (PLEG): container finished" podID="8b29d3b2-0957-46a6-a304-c085cbf41fb2" containerID="ca57b1d0d4dab17a3c09f6eb7423a49dba5582ad45ce2f2e76ee165762892f32" exitCode=0
Jul 28 11:15:01.676120 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:01.676053    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz" event={"ID":"8b29d3b2-0957-46a6-a304-c085cbf41fb2","Type":"ContainerDied","Data":"ca57b1d0d4dab17a3c09f6eb7423a49dba5582ad45ce2f2e76ee165762892f32"}
Jul 28 11:15:02.840817 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:02.840787    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"
Jul 28 11:15:02.844192 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:02.844155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"
Jul 28 11:15:03.687007 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:03.686980    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v"
Jul 28 11:15:03.687176 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:03.686985    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753955-gcj7v" event={"ID":"b25fdb20-b2a6-489a-9dd3-7aaf067c9508","Type":"ContainerDied","Data":"5d96dde903e5155d0af8c5a5751a7de9522b5dbabccf327cd7ba3b4dabf7ed48"}
Jul 28 11:15:03.687176 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:03.687104    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d96dde903e5155d0af8c5a5751a7de9522b5dbabccf327cd7ba3b4dabf7ed48"
Jul 28 11:15:03.688392 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:03.688375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"
Jul 28 11:15:03.688510 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:03.688398    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz" event={"ID":"8b29d3b2-0957-46a6-a304-c085cbf41fb2","Type":"ContainerDied","Data":"12d06fdde5a9dbdeda99b734577d777d753a5e7f01c8eac332bffed94d6825fa"}
Jul 28 11:15:03.688510 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:03.688447    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12d06fdde5a9dbdeda99b734577d777d753a5e7f01c8eac332bffed94d6825fa"
Jul 28 11:15:27.221713 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:27.221673    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"]
Jul 28 11:15:30.228302 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:30.228256    2578 scope.go:117] "RemoveContainer" containerID="9a056a97e51c2c8d9f2f4ddc63e892703013a20e2c4da0479ecc19adeae493c6"
Jul 28 11:15:42.241305 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.241226    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4" podUID="c9345a0f-7747-49b7-8fd6-b0aab32f518d" containerName="main" containerID="cri-o://c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db" gracePeriod=45
Jul 28 11:15:42.487229 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.487200    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:15:42.560696 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560607    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tmp-dir\") pod \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") "
Jul 28 11:15:42.560696 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560660    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-model-cache\") pod \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") "
Jul 28 11:15:42.560696 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560691    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-home\") pod \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") "
Jul 28 11:15:42.560967 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-dshm\") pod \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") "
Jul 28 11:15:42.560967 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77p5n\" (UniqueName: \"kubernetes.io/projected/c9345a0f-7747-49b7-8fd6-b0aab32f518d-kube-api-access-77p5n\") pod \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") "
Jul 28 11:15:42.560967 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tls-certs\") pod \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\" (UID: \"c9345a0f-7747-49b7-8fd6-b0aab32f518d\") "
Jul 28 11:15:42.560967 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.560940    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9345a0f-7747-49b7-8fd6-b0aab32f518d" (UID: "c9345a0f-7747-49b7-8fd6-b0aab32f518d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:15:42.561187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.561032    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-home" (OuterVolumeSpecName: "home") pod "c9345a0f-7747-49b7-8fd6-b0aab32f518d" (UID: "c9345a0f-7747-49b7-8fd6-b0aab32f518d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:15:42.561187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.561045    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9345a0f-7747-49b7-8fd6-b0aab32f518d" (UID: "c9345a0f-7747-49b7-8fd6-b0aab32f518d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:15:42.561187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.561123    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:15:42.563167 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.563136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9345a0f-7747-49b7-8fd6-b0aab32f518d" (UID: "c9345a0f-7747-49b7-8fd6-b0aab32f518d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:15:42.563167 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.563160    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-dshm" (OuterVolumeSpecName: "dshm") pod "c9345a0f-7747-49b7-8fd6-b0aab32f518d" (UID: "c9345a0f-7747-49b7-8fd6-b0aab32f518d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:15:42.563362 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.563236    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9345a0f-7747-49b7-8fd6-b0aab32f518d-kube-api-access-77p5n" (OuterVolumeSpecName: "kube-api-access-77p5n") pod "c9345a0f-7747-49b7-8fd6-b0aab32f518d" (UID: "c9345a0f-7747-49b7-8fd6-b0aab32f518d"). InnerVolumeSpecName "kube-api-access-77p5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:15:42.662334 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.662299    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:15:42.662334 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.662331    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:15:42.662334 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.662339    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9345a0f-7747-49b7-8fd6-b0aab32f518d-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:15:42.662580 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.662348    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77p5n\" (UniqueName: \"kubernetes.io/projected/c9345a0f-7747-49b7-8fd6-b0aab32f518d-kube-api-access-77p5n\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:15:42.662580 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.662358    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9345a0f-7747-49b7-8fd6-b0aab32f518d-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:15:42.876591 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.876499    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9345a0f-7747-49b7-8fd6-b0aab32f518d" containerID="c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db" exitCode=0
Jul 28 11:15:42.876591 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.876545    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"
Jul 28 11:15:42.876591 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.876566    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4" event={"ID":"c9345a0f-7747-49b7-8fd6-b0aab32f518d","Type":"ContainerDied","Data":"c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db"}
Jul 28 11:15:42.876834 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.876604    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4" event={"ID":"c9345a0f-7747-49b7-8fd6-b0aab32f518d","Type":"ContainerDied","Data":"c07cd87687a8b89ddbc68bc84093573c5c5ac80b1543c3b8f67f4e5a76a52ae4"}
Jul 28 11:15:42.876834 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.876626    2578 scope.go:117] "RemoveContainer" containerID="c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db"
Jul 28 11:15:42.885764 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.885742    2578 scope.go:117] "RemoveContainer" containerID="c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db"
Jul 28 11:15:42.886034 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:15:42.886017    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db\": container with ID starting with c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db not found: ID does not exist" containerID="c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db"
Jul 28 11:15:42.886085 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.886044    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db"} err="failed to get container status \"c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db\": rpc error: code = NotFound desc = could not find container \"c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db\": container with ID starting with c49b569700f317ceefedeae3f7c4b2271b32d8b9b64036a1e0a1f91c681c30db not found: ID does not exist"
Jul 28 11:15:42.897027 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.896990    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"]
Jul 28 11:15:42.902636 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:42.902602    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/auth-test-model-4cda06-kserve-6847674875-stww4"]
Jul 28 11:15:43.500156 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:43.500117    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9345a0f-7747-49b7-8fd6-b0aab32f518d" path="/var/lib/kubelet/pods/c9345a0f-7747-49b7-8fd6-b0aab32f518d/volumes"
Jul 28 11:15:45.262048 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262015    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"]
Jul 28 11:15:45.262561 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262539    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b25fdb20-b2a6-489a-9dd3-7aaf067c9508" containerName="cleanup"
Jul 28 11:15:45.262561 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262560    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b25fdb20-b2a6-489a-9dd3-7aaf067c9508" containerName="cleanup"
Jul 28 11:15:45.262692 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262587    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b29d3b2-0957-46a6-a304-c085cbf41fb2" containerName="cleanup"
Jul 28 11:15:45.262692 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262596    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b29d3b2-0957-46a6-a304-c085cbf41fb2" containerName="cleanup"
Jul 28 11:15:45.262692 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9345a0f-7747-49b7-8fd6-b0aab32f518d" containerName="main"
Jul 28 11:15:45.262692 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262630    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9345a0f-7747-49b7-8fd6-b0aab32f518d" containerName="main"
Jul 28 11:15:45.262884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262730    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b25fdb20-b2a6-489a-9dd3-7aaf067c9508" containerName="cleanup"
Jul 28 11:15:45.262884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262747    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b29d3b2-0957-46a6-a304-c085cbf41fb2" containerName="cleanup"
Jul 28 11:15:45.262884 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.262760    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9345a0f-7747-49b7-8fd6-b0aab32f518d" containerName="main"
Jul 28 11:15:45.267329 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.267305    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.269859 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.269833    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"openshift-service-ca.crt\""
Jul 28 11:15:45.269976 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.269874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"kube-root-ca.crt\""
Jul 28 11:15:45.270707 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.270688    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"default-dockercfg-rffrz\""
Jul 28 11:15:45.270801 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.270712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"sub-test-model-4cda06-kserve-self-signed-certs\""
Jul 28 11:15:45.273927 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.273900    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"]
Jul 28 11:15:45.386638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.386592    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-dshm\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.386638 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.386642    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc8m\" (UniqueName: \"kubernetes.io/projected/e62308a7-a115-4aec-b37a-37d5a903fc31-kube-api-access-nmc8m\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.386870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.386729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-model-cache\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.386870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.386762    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-home\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.386870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.386782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-tmp-dir\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.386870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.386843    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62308a7-a115-4aec-b37a-37d5a903fc31-tls-certs\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.487890 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.487852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62308a7-a115-4aec-b37a-37d5a903fc31-tls-certs\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.487914    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-dshm\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.487935    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc8m\" (UniqueName: \"kubernetes.io/projected/e62308a7-a115-4aec-b37a-37d5a903fc31-kube-api-access-nmc8m\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.487973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-model-cache\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.487995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-home\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.488012    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-tmp-dir\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488492 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.488469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-home\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488492 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.488485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-tmp-dir\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.488617 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.488535    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-model-cache\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.490484 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.490458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-dshm\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.490721 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.490705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62308a7-a115-4aec-b37a-37d5a903fc31-tls-certs\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.497495 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.497459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc8m\" (UniqueName: \"kubernetes.io/projected/e62308a7-a115-4aec-b37a-37d5a903fc31-kube-api-access-nmc8m\") pod \"sub-test-model-4cda06-kserve-5f747fc679-p7xcw\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") " pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.578593 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.578481    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:45.717897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.717870    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"]
Jul 28 11:15:45.720143 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:15:45.720103    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode62308a7_a115_4aec_b37a_37d5a903fc31.slice/crio-564b31e7d15651bad833deb6d115ff3e75514f84df4e8d6b2e7136d98516de2c WatchSource:0}: Error finding container 564b31e7d15651bad833deb6d115ff3e75514f84df4e8d6b2e7136d98516de2c: Status 404 returned error can't find the container with id 564b31e7d15651bad833deb6d115ff3e75514f84df4e8d6b2e7136d98516de2c
Jul 28 11:15:45.893411 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.893305    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw" event={"ID":"e62308a7-a115-4aec-b37a-37d5a903fc31","Type":"ContainerStarted","Data":"54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96"}
Jul 28 11:15:45.893411 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.893356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw" event={"ID":"e62308a7-a115-4aec-b37a-37d5a903fc31","Type":"ContainerStarted","Data":"564b31e7d15651bad833deb6d115ff3e75514f84df4e8d6b2e7136d98516de2c"}
Jul 28 11:15:45.909359 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:45.909308    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw" podStartSLOduration=0.909291905 podStartE2EDuration="909.291905ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:15:45.907597462 +0000 UTC m=+3136.949689616" watchObservedRunningTime="2026-07-28 11:15:45.909291905 +0000 UTC m=+3136.951384057"
Jul 28 11:15:55.578906 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:55.578804    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:55.578906 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:55.578877    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:55.591746 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:55.591715    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:15:55.959921 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:15:55.959879    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:17:29.530182 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:29.530096    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"]
Jul 28 11:17:44.550221 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.550171    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw" podUID="e62308a7-a115-4aec-b37a-37d5a903fc31" containerName="main" containerID="cri-o://54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96" gracePeriod=45
Jul 28 11:17:44.815118 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.815035    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:17:44.851899 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.851856    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-dshm\") pod \"e62308a7-a115-4aec-b37a-37d5a903fc31\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") "
Jul 28 11:17:44.851899 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.851898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-home\") pod \"e62308a7-a115-4aec-b37a-37d5a903fc31\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") "
Jul 28 11:17:44.852162 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.851930    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmc8m\" (UniqueName: \"kubernetes.io/projected/e62308a7-a115-4aec-b37a-37d5a903fc31-kube-api-access-nmc8m\") pod \"e62308a7-a115-4aec-b37a-37d5a903fc31\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") "
Jul 28 11:17:44.852162 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.851979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62308a7-a115-4aec-b37a-37d5a903fc31-tls-certs\") pod \"e62308a7-a115-4aec-b37a-37d5a903fc31\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") "
Jul 28 11:17:44.852162 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.852024    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-tmp-dir\") pod \"e62308a7-a115-4aec-b37a-37d5a903fc31\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") "
Jul 28 11:17:44.852162 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.852060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-model-cache\") pod \"e62308a7-a115-4aec-b37a-37d5a903fc31\" (UID: \"e62308a7-a115-4aec-b37a-37d5a903fc31\") "
Jul 28 11:17:44.852410 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.852235    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-home" (OuterVolumeSpecName: "home") pod "e62308a7-a115-4aec-b37a-37d5a903fc31" (UID: "e62308a7-a115-4aec-b37a-37d5a903fc31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:17:44.852410 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.852347    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:17:44.852514 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.852379    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e62308a7-a115-4aec-b37a-37d5a903fc31" (UID: "e62308a7-a115-4aec-b37a-37d5a903fc31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:17:44.852514 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.852426    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-model-cache" (OuterVolumeSpecName: "model-cache") pod "e62308a7-a115-4aec-b37a-37d5a903fc31" (UID: "e62308a7-a115-4aec-b37a-37d5a903fc31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:17:44.854408 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.854374    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e62308a7-a115-4aec-b37a-37d5a903fc31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e62308a7-a115-4aec-b37a-37d5a903fc31" (UID: "e62308a7-a115-4aec-b37a-37d5a903fc31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:17:44.854705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.854681    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-dshm" (OuterVolumeSpecName: "dshm") pod "e62308a7-a115-4aec-b37a-37d5a903fc31" (UID: "e62308a7-a115-4aec-b37a-37d5a903fc31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:17:44.854705 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.854696    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e62308a7-a115-4aec-b37a-37d5a903fc31-kube-api-access-nmc8m" (OuterVolumeSpecName: "kube-api-access-nmc8m") pod "e62308a7-a115-4aec-b37a-37d5a903fc31" (UID: "e62308a7-a115-4aec-b37a-37d5a903fc31"). InnerVolumeSpecName "kube-api-access-nmc8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:17:44.953219 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.953179    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:17:44.953219 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.953214    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmc8m\" (UniqueName: \"kubernetes.io/projected/e62308a7-a115-4aec-b37a-37d5a903fc31-kube-api-access-nmc8m\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:17:44.953219 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.953224    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e62308a7-a115-4aec-b37a-37d5a903fc31-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:17:44.953484 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.953235    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:17:44.953484 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:44.953243    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e62308a7-a115-4aec-b37a-37d5a903fc31-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:17:45.487406 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.487368    2578 generic.go:356] "Generic (PLEG): container finished" podID="e62308a7-a115-4aec-b37a-37d5a903fc31" containerID="54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96" exitCode=0
Jul 28 11:17:45.487608 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.487415    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"
Jul 28 11:17:45.487608 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.487453    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw" event={"ID":"e62308a7-a115-4aec-b37a-37d5a903fc31","Type":"ContainerDied","Data":"54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96"}
Jul 28 11:17:45.487608 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.487498    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw" event={"ID":"e62308a7-a115-4aec-b37a-37d5a903fc31","Type":"ContainerDied","Data":"564b31e7d15651bad833deb6d115ff3e75514f84df4e8d6b2e7136d98516de2c"}
Jul 28 11:17:45.487608 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.487517    2578 scope.go:117] "RemoveContainer" containerID="54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96"
Jul 28 11:17:45.504610 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.504585    2578 scope.go:117] "RemoveContainer" containerID="54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96"
Jul 28 11:17:45.504961 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:17:45.504942    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96\": container with ID starting with 54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96 not found: ID does not exist" containerID="54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96"
Jul 28 11:17:45.505020 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.504968    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96"} err="failed to get container status \"54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96\": rpc error: code = NotFound desc = could not find container \"54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96\": container with ID starting with 54392e9103bacfa5c5dd22c5327219d46e45647b6fdc9cd28447dda8671b6f96 not found: ID does not exist"
Jul 28 11:17:45.517313 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.517255    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"]
Jul 28 11:17:45.521598 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:45.521563    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/sub-test-model-4cda06-kserve-5f747fc679-p7xcw"]
Jul 28 11:17:47.495746 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:47.495700    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e62308a7-a115-4aec-b37a-37d5a903fc31" path="/var/lib/kubelet/pods/e62308a7-a115-4aec-b37a-37d5a903fc31/volumes"
Jul 28 11:17:49.589841 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.589798    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"]
Jul 28 11:17:49.590233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.590139    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e62308a7-a115-4aec-b37a-37d5a903fc31" containerName="main"
Jul 28 11:17:49.590233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.590149    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e62308a7-a115-4aec-b37a-37d5a903fc31" containerName="main"
Jul 28 11:17:49.590233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.590214    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e62308a7-a115-4aec-b37a-37d5a903fc31" containerName="main"
Jul 28 11:17:49.594447 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.594424    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.597558 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.597533    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"kube-root-ca.crt\""
Jul 28 11:17:49.597707 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.597653    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"rate-test-model-4cda06-kserve-self-signed-certs\""
Jul 28 11:17:49.598817 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.598796    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"default-dockercfg-rffrz\""
Jul 28 11:17:49.598933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.598799    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-4cda06\"/\"openshift-service-ca.crt\""
Jul 28 11:17:49.606323 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.606263    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"]
Jul 28 11:17:49.696223 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.696181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tmp-dir\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.696223 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.696233    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-model-cache\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.696477 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.696257    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-home\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.696477 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.696386    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-dshm\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.696477 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.696414    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tls-certs\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.696477 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.696456    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkwcd\" (UniqueName: \"kubernetes.io/projected/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-kube-api-access-rkwcd\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797371 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797333    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-model-cache\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797555 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-home\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797555 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797452    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-dshm\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797555 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tls-certs\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797555 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkwcd\" (UniqueName: \"kubernetes.io/projected/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-kube-api-access-rkwcd\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797765 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tmp-dir\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797819 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-model-cache\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797895 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-home\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.797951 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.797920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tmp-dir\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.799915 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.799891    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-dshm\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.800169 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.800151    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tls-certs\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.809173 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.809138    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkwcd\" (UniqueName: \"kubernetes.io/projected/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-kube-api-access-rkwcd\") pod \"rate-test-model-4cda06-kserve-686754c-jrcbt\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") " pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:49.903931 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:49.903830    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:50.050610 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:50.050581    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"]
Jul 28 11:17:50.052732 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:17:50.052704    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc524d3b_5bc1_49af_96e0_35060e9e1fc7.slice/crio-c9c7ea1b659d35b66a50188926d6329b64e794de98df676424c57199aaac0076 WatchSource:0}: Error finding container c9c7ea1b659d35b66a50188926d6329b64e794de98df676424c57199aaac0076: Status 404 returned error can't find the container with id c9c7ea1b659d35b66a50188926d6329b64e794de98df676424c57199aaac0076
Jul 28 11:17:50.516239 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:50.516192    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt" event={"ID":"cc524d3b-5bc1-49af-96e0-35060e9e1fc7","Type":"ContainerStarted","Data":"3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559"}
Jul 28 11:17:50.516448 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:50.516252    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt" event={"ID":"cc524d3b-5bc1-49af-96e0-35060e9e1fc7","Type":"ContainerStarted","Data":"c9c7ea1b659d35b66a50188926d6329b64e794de98df676424c57199aaac0076"}
Jul 28 11:17:50.533927 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:50.533868    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt" podStartSLOduration=1.533850713 podStartE2EDuration="1.533850713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:17:50.532322742 +0000 UTC m=+3261.574414910" watchObservedRunningTime="2026-07-28 11:17:50.533850713 +0000 UTC m=+3261.575942865"
Jul 28 11:17:59.904702 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:59.904652    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:59.904702 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:59.904710    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:17:59.917950 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:17:59.917918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:18:00.574420 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:18:00.574391    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:18:30.060554 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:18:30.060520    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:18:30.070956 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:18:30.070928    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:19:31.186065 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:31.186022    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"]
Jul 28 11:19:34.020132 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.020087    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6975-140712451002160"]
Jul 28 11:19:34.024692 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.024669    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:34.027661 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.027631    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l49gt\""
Jul 28 11:19:34.029860 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.029834    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451002160"]
Jul 28 11:19:34.149629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.149585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjzt6\" (UniqueName: \"kubernetes.io/projected/e29d9804-0d83-49ec-aba2-bf01f248cf73-kube-api-access-hjzt6\") pod \"test-curl-6975-140712451002160\" (UID: \"e29d9804-0d83-49ec-aba2-bf01f248cf73\") " pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:34.250124 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.250079    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjzt6\" (UniqueName: \"kubernetes.io/projected/e29d9804-0d83-49ec-aba2-bf01f248cf73-kube-api-access-hjzt6\") pod \"test-curl-6975-140712451002160\" (UID: \"e29d9804-0d83-49ec-aba2-bf01f248cf73\") " pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:34.258891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.258854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjzt6\" (UniqueName: \"kubernetes.io/projected/e29d9804-0d83-49ec-aba2-bf01f248cf73-kube-api-access-hjzt6\") pod \"test-curl-6975-140712451002160\" (UID: \"e29d9804-0d83-49ec-aba2-bf01f248cf73\") " pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:34.335106 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.335010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:34.474748 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.474723    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451002160"]
Jul 28 11:19:34.476967 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:19:34.476935    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode29d9804_0d83_49ec_aba2_bf01f248cf73.slice/crio-af940b3b470216e6a50255da8d3135b89a1a116d6e2eac8ef51e25db2be8cca1 WatchSource:0}: Error finding container af940b3b470216e6a50255da8d3135b89a1a116d6e2eac8ef51e25db2be8cca1: Status 404 returned error can't find the container with id af940b3b470216e6a50255da8d3135b89a1a116d6e2eac8ef51e25db2be8cca1
Jul 28 11:19:34.478148 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:34.478131    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:19:35.033084 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:35.033043    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451002160" event={"ID":"e29d9804-0d83-49ec-aba2-bf01f248cf73","Type":"ContainerStarted","Data":"af940b3b470216e6a50255da8d3135b89a1a116d6e2eac8ef51e25db2be8cca1"}
Jul 28 11:19:36.039729 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:36.039691    2578 generic.go:356] "Generic (PLEG): container finished" podID="e29d9804-0d83-49ec-aba2-bf01f248cf73" containerID="6fb613eeb2739ec00b817b34dfece5af9eb0a4f24d2ca5a7174a508fc696e9f2" exitCode=0
Jul 28 11:19:36.040187 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:36.039778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451002160" event={"ID":"e29d9804-0d83-49ec-aba2-bf01f248cf73","Type":"ContainerDied","Data":"6fb613eeb2739ec00b817b34dfece5af9eb0a4f24d2ca5a7174a508fc696e9f2"}
Jul 28 11:19:37.183382 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.183353    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:37.230457 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.230423    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6975-140712451002160_e29d9804-0d83-49ec-aba2-bf01f248cf73/test-curl-6975-140712451002160/0.log"
Jul 28 11:19:37.252979 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.252936    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6975-140712451002160"]
Jul 28 11:19:37.256843 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.256809    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6975-140712451002160"]
Jul 28 11:19:37.281052 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.281012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjzt6\" (UniqueName: \"kubernetes.io/projected/e29d9804-0d83-49ec-aba2-bf01f248cf73-kube-api-access-hjzt6\") pod \"e29d9804-0d83-49ec-aba2-bf01f248cf73\" (UID: \"e29d9804-0d83-49ec-aba2-bf01f248cf73\") "
Jul 28 11:19:37.283375 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.283338    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29d9804-0d83-49ec-aba2-bf01f248cf73-kube-api-access-hjzt6" (OuterVolumeSpecName: "kube-api-access-hjzt6") pod "e29d9804-0d83-49ec-aba2-bf01f248cf73" (UID: "e29d9804-0d83-49ec-aba2-bf01f248cf73"). InnerVolumeSpecName "kube-api-access-hjzt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:19:37.382223 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.382128    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjzt6\" (UniqueName: \"kubernetes.io/projected/e29d9804-0d83-49ec-aba2-bf01f248cf73-kube-api-access-hjzt6\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:37.394940 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.394897    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6975-140712450998832"]
Jul 28 11:19:37.395326 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.395312    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e29d9804-0d83-49ec-aba2-bf01f248cf73" containerName="test-curl-6975-140712451002160"
Jul 28 11:19:37.395389 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.395328    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e29d9804-0d83-49ec-aba2-bf01f248cf73" containerName="test-curl-6975-140712451002160"
Jul 28 11:19:37.395435 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.395395    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e29d9804-0d83-49ec-aba2-bf01f248cf73" containerName="test-curl-6975-140712451002160"
Jul 28 11:19:37.398369 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.398352    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:37.402063 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.402038    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712450998832"]
Jul 28 11:19:37.482991 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.482948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmlwr\" (UniqueName: \"kubernetes.io/projected/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d-kube-api-access-tmlwr\") pod \"test-curl-6975-140712450998832\" (UID: \"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d\") " pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:37.494820 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.494787    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e29d9804-0d83-49ec-aba2-bf01f248cf73" path="/var/lib/kubelet/pods/e29d9804-0d83-49ec-aba2-bf01f248cf73/volumes"
Jul 28 11:19:37.584204 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.584151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmlwr\" (UniqueName: \"kubernetes.io/projected/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d-kube-api-access-tmlwr\") pod \"test-curl-6975-140712450998832\" (UID: \"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d\") " pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:37.593522 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.593494    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmlwr\" (UniqueName: \"kubernetes.io/projected/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d-kube-api-access-tmlwr\") pod \"test-curl-6975-140712450998832\" (UID: \"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d\") " pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:37.709029 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.708979    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:37.849423 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:37.849392    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712450998832"]
Jul 28 11:19:37.851295 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:19:37.851251    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb85eebc7_e34f_4a6b_8f06_b3dab7027c8d.slice/crio-21b390321a4c87c928217d9edfb464f8d75a2a29fe0f03a3cbb9f9ef2c4d1e1d WatchSource:0}: Error finding container 21b390321a4c87c928217d9edfb464f8d75a2a29fe0f03a3cbb9f9ef2c4d1e1d: Status 404 returned error can't find the container with id 21b390321a4c87c928217d9edfb464f8d75a2a29fe0f03a3cbb9f9ef2c4d1e1d
Jul 28 11:19:38.051513 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:38.051415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712450998832" event={"ID":"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d","Type":"ContainerStarted","Data":"21b390321a4c87c928217d9edfb464f8d75a2a29fe0f03a3cbb9f9ef2c4d1e1d"}
Jul 28 11:19:38.052782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:38.052760    2578 scope.go:117] "RemoveContainer" containerID="6fb613eeb2739ec00b817b34dfece5af9eb0a4f24d2ca5a7174a508fc696e9f2"
Jul 28 11:19:38.052946 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:38.052765    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451002160"
Jul 28 11:19:39.059450 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:39.059414    2578 generic.go:356] "Generic (PLEG): container finished" podID="b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" containerID="48e87d9a9e490725de78aefaefb011042749018ebd5860d087072401fe43272c" exitCode=0
Jul 28 11:19:39.060005 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:39.059508    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712450998832" event={"ID":"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d","Type":"ContainerDied","Data":"48e87d9a9e490725de78aefaefb011042749018ebd5860d087072401fe43272c"}
Jul 28 11:19:40.201905 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.201879    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:40.220856 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.220825    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6975-140712450998832_b85eebc7-e34f-4a6b-8f06-b3dab7027c8d/test-curl-6975-140712450998832/0.log"
Jul 28 11:19:40.250057 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.250030    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6975-140712450998832"]
Jul 28 11:19:40.256210 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.256179    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6975-140712450998832"]
Jul 28 11:19:40.312897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.312855    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tmlwr\" (UniqueName: \"kubernetes.io/projected/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d-kube-api-access-tmlwr\") pod \"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d\" (UID: \"b85eebc7-e34f-4a6b-8f06-b3dab7027c8d\") "
Jul 28 11:19:40.315128 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.315102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d-kube-api-access-tmlwr" (OuterVolumeSpecName: "kube-api-access-tmlwr") pod "b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" (UID: "b85eebc7-e34f-4a6b-8f06-b3dab7027c8d"). InnerVolumeSpecName "kube-api-access-tmlwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:19:40.414158 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.414116    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmlwr\" (UniqueName: \"kubernetes.io/projected/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d-kube-api-access-tmlwr\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:40.438929 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.438886    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6975-140712451110928"]
Jul 28 11:19:40.439255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.439242    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" containerName="test-curl-6975-140712450998832"
Jul 28 11:19:40.439324 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.439258    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" containerName="test-curl-6975-140712450998832"
Jul 28 11:19:40.439362 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.439338    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" containerName="test-curl-6975-140712450998832"
Jul 28 11:19:40.442289 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.442248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:40.449388 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.449354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451110928"]
Jul 28 11:19:40.515184 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.515145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ckwt\" (UniqueName: \"kubernetes.io/projected/87eea528-2bd3-4cba-8b8a-9c476d9f19d0-kube-api-access-2ckwt\") pod \"test-curl-6975-140712451110928\" (UID: \"87eea528-2bd3-4cba-8b8a-9c476d9f19d0\") " pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:40.616401 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.616299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ckwt\" (UniqueName: \"kubernetes.io/projected/87eea528-2bd3-4cba-8b8a-9c476d9f19d0-kube-api-access-2ckwt\") pod \"test-curl-6975-140712451110928\" (UID: \"87eea528-2bd3-4cba-8b8a-9c476d9f19d0\") " pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:40.624560 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.624525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ckwt\" (UniqueName: \"kubernetes.io/projected/87eea528-2bd3-4cba-8b8a-9c476d9f19d0-kube-api-access-2ckwt\") pod \"test-curl-6975-140712451110928\" (UID: \"87eea528-2bd3-4cba-8b8a-9c476d9f19d0\") " pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:40.752738 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.752693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:40.890112 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:40.890085    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451110928"]
Jul 28 11:19:40.892223 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:19:40.892192    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod87eea528_2bd3_4cba_8b8a_9c476d9f19d0.slice/crio-7b2b059efc5266bd9412386a0ce79771f770c27d05c11bd70742b4d8d32d72a6 WatchSource:0}: Error finding container 7b2b059efc5266bd9412386a0ce79771f770c27d05c11bd70742b4d8d32d72a6: Status 404 returned error can't find the container with id 7b2b059efc5266bd9412386a0ce79771f770c27d05c11bd70742b4d8d32d72a6
Jul 28 11:19:41.079041 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:41.078991    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451110928" event={"ID":"87eea528-2bd3-4cba-8b8a-9c476d9f19d0","Type":"ContainerStarted","Data":"7b2b059efc5266bd9412386a0ce79771f770c27d05c11bd70742b4d8d32d72a6"}
Jul 28 11:19:41.080466 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:41.080435    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b390321a4c87c928217d9edfb464f8d75a2a29fe0f03a3cbb9f9ef2c4d1e1d"
Jul 28 11:19:41.080594 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:41.080527    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:41.091023 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:19:41.090986    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6975-140712450998832\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" pod="opendatahub/test-curl-6975-140712450998832"
Jul 28 11:19:41.494950 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:41.494866    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b85eebc7-e34f-4a6b-8f06-b3dab7027c8d" path="/var/lib/kubelet/pods/b85eebc7-e34f-4a6b-8f06-b3dab7027c8d/volumes"
Jul 28 11:19:42.086887 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:42.086851    2578 generic.go:356] "Generic (PLEG): container finished" podID="87eea528-2bd3-4cba-8b8a-9c476d9f19d0" containerID="3190bec414b58eed8b3f6f22f4109ee7391de9ff9ce739e23d62546b6bea52c1" exitCode=0
Jul 28 11:19:42.087104 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:42.086938    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451110928" event={"ID":"87eea528-2bd3-4cba-8b8a-9c476d9f19d0","Type":"ContainerDied","Data":"3190bec414b58eed8b3f6f22f4109ee7391de9ff9ce739e23d62546b6bea52c1"}
Jul 28 11:19:43.233809 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.233782    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:43.253380 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.253350    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6975-140712451110928_87eea528-2bd3-4cba-8b8a-9c476d9f19d0/test-curl-6975-140712451110928/0.log"
Jul 28 11:19:43.280327 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.280261    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6975-140712451110928"]
Jul 28 11:19:43.283704 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.283675    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6975-140712451110928"]
Jul 28 11:19:43.343610 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.343535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ckwt\" (UniqueName: \"kubernetes.io/projected/87eea528-2bd3-4cba-8b8a-9c476d9f19d0-kube-api-access-2ckwt\") pod \"87eea528-2bd3-4cba-8b8a-9c476d9f19d0\" (UID: \"87eea528-2bd3-4cba-8b8a-9c476d9f19d0\") "
Jul 28 11:19:43.345861 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.345832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87eea528-2bd3-4cba-8b8a-9c476d9f19d0-kube-api-access-2ckwt" (OuterVolumeSpecName: "kube-api-access-2ckwt") pod "87eea528-2bd3-4cba-8b8a-9c476d9f19d0" (UID: "87eea528-2bd3-4cba-8b8a-9c476d9f19d0"). InnerVolumeSpecName "kube-api-access-2ckwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:19:43.444552 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.444513    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ckwt\" (UniqueName: \"kubernetes.io/projected/87eea528-2bd3-4cba-8b8a-9c476d9f19d0-kube-api-access-2ckwt\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:43.494750 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:43.494717    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87eea528-2bd3-4cba-8b8a-9c476d9f19d0" path="/var/lib/kubelet/pods/87eea528-2bd3-4cba-8b8a-9c476d9f19d0/volumes"
Jul 28 11:19:44.098812 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:44.098786    2578 scope.go:117] "RemoveContainer" containerID="3190bec414b58eed8b3f6f22f4109ee7391de9ff9ce739e23d62546b6bea52c1"
Jul 28 11:19:44.098812 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:44.098806    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451110928"
Jul 28 11:19:46.206488 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.206441    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt" podUID="cc524d3b-5bc1-49af-96e0-35060e9e1fc7" containerName="main" containerID="cri-o://3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559" gracePeriod=45
Jul 28 11:19:46.340876 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.340833    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:46.341410 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.341394    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87eea528-2bd3-4cba-8b8a-9c476d9f19d0" containerName="test-curl-6975-140712451110928"
Jul 28 11:19:46.341465 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.341414    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="87eea528-2bd3-4cba-8b8a-9c476d9f19d0" containerName="test-curl-6975-140712451110928"
Jul 28 11:19:46.341524 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.341512    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="87eea528-2bd3-4cba-8b8a-9c476d9f19d0" containerName="test-curl-6975-140712451110928"
Jul 28 11:19:46.344883 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.344861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:46.347684 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.347661    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-l49gt\""
Jul 28 11:19:46.350814 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.350787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:46.471716 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.471637    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:19:46.472694 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.472672    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrjx\" (UniqueName: \"kubernetes.io/projected/e134db85-15f0-4354-bee2-5019e6778480-kube-api-access-mrrjx\") pod \"test-curl-6975-140712451111936\" (UID: \"e134db85-15f0-4354-bee2-5019e6778480\") " pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:46.573387 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573352    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tls-certs\") pod \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") "
Jul 28 11:19:46.573565 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-home\") pod \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") "
Jul 28 11:19:46.573565 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkwcd\" (UniqueName: \"kubernetes.io/projected/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-kube-api-access-rkwcd\") pod \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") "
Jul 28 11:19:46.573565 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573542    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tmp-dir\") pod \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") "
Jul 28 11:19:46.573693 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573571    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-model-cache\") pod \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") "
Jul 28 11:19:46.573693 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-dshm\") pod \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\" (UID: \"cc524d3b-5bc1-49af-96e0-35060e9e1fc7\") "
Jul 28 11:19:46.573798 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573752    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-home" (OuterVolumeSpecName: "home") pod "cc524d3b-5bc1-49af-96e0-35060e9e1fc7" (UID: "cc524d3b-5bc1-49af-96e0-35060e9e1fc7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:19:46.573881 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573849    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-model-cache" (OuterVolumeSpecName: "model-cache") pod "cc524d3b-5bc1-49af-96e0-35060e9e1fc7" (UID: "cc524d3b-5bc1-49af-96e0-35060e9e1fc7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:19:46.573938 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573878    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrjx\" (UniqueName: \"kubernetes.io/projected/e134db85-15f0-4354-bee2-5019e6778480-kube-api-access-mrrjx\") pod \"test-curl-6975-140712451111936\" (UID: \"e134db85-15f0-4354-bee2-5019e6778480\") " pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:46.573993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573948    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:46.573993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573964    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:46.573993 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.573856    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cc524d3b-5bc1-49af-96e0-35060e9e1fc7" (UID: "cc524d3b-5bc1-49af-96e0-35060e9e1fc7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:19:46.575824 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.575792    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-dshm" (OuterVolumeSpecName: "dshm") pod "cc524d3b-5bc1-49af-96e0-35060e9e1fc7" (UID: "cc524d3b-5bc1-49af-96e0-35060e9e1fc7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:19:46.575930 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.575818    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cc524d3b-5bc1-49af-96e0-35060e9e1fc7" (UID: "cc524d3b-5bc1-49af-96e0-35060e9e1fc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:19:46.575930 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.575832    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-kube-api-access-rkwcd" (OuterVolumeSpecName: "kube-api-access-rkwcd") pod "cc524d3b-5bc1-49af-96e0-35060e9e1fc7" (UID: "cc524d3b-5bc1-49af-96e0-35060e9e1fc7"). InnerVolumeSpecName "kube-api-access-rkwcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:19:46.582142 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.582116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrjx\" (UniqueName: \"kubernetes.io/projected/e134db85-15f0-4354-bee2-5019e6778480-kube-api-access-mrrjx\") pod \"test-curl-6975-140712451111936\" (UID: \"e134db85-15f0-4354-bee2-5019e6778480\") " pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:46.656422 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.656383    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:46.674801 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.674759    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkwcd\" (UniqueName: \"kubernetes.io/projected/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-kube-api-access-rkwcd\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:46.674801 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.674794    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:46.674801 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.674804    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:46.675030 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.674812    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cc524d3b-5bc1-49af-96e0-35060e9e1fc7-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:46.794079 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:46.794039    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:46.797220 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:19:46.797186    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode134db85_15f0_4354_bee2_5019e6778480.slice/crio-f094c1cc41790b19bd2d9223619900e71ea6171aa7b2d4dd835908bc64aeeb9c WatchSource:0}: Error finding container f094c1cc41790b19bd2d9223619900e71ea6171aa7b2d4dd835908bc64aeeb9c: Status 404 returned error can't find the container with id f094c1cc41790b19bd2d9223619900e71ea6171aa7b2d4dd835908bc64aeeb9c
Jul 28 11:19:47.115826 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.115795    2578 generic.go:356] "Generic (PLEG): container finished" podID="cc524d3b-5bc1-49af-96e0-35060e9e1fc7" containerID="3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559" exitCode=0
Jul 28 11:19:47.115937 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.115887    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"
Jul 28 11:19:47.115937 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.115903    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt" event={"ID":"cc524d3b-5bc1-49af-96e0-35060e9e1fc7","Type":"ContainerDied","Data":"3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559"}
Jul 28 11:19:47.116049 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.115944    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt" event={"ID":"cc524d3b-5bc1-49af-96e0-35060e9e1fc7","Type":"ContainerDied","Data":"c9c7ea1b659d35b66a50188926d6329b64e794de98df676424c57199aaac0076"}
Jul 28 11:19:47.116049 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.115965    2578 scope.go:117] "RemoveContainer" containerID="3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559"
Jul 28 11:19:47.117393 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.117364    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451111936" event={"ID":"e134db85-15f0-4354-bee2-5019e6778480","Type":"ContainerStarted","Data":"f094c1cc41790b19bd2d9223619900e71ea6171aa7b2d4dd835908bc64aeeb9c"}
Jul 28 11:19:47.125811 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.125783    2578 scope.go:117] "RemoveContainer" containerID="3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559"
Jul 28 11:19:47.126173 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:19:47.126151    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559\": container with ID starting with 3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559 not found: ID does not exist" containerID="3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559"
Jul 28 11:19:47.126250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.126184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559"} err="failed to get container status \"3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559\": rpc error: code = NotFound desc = could not find container \"3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559\": container with ID starting with 3fcc084c800796428dc6ad6b009ad420d4a2a3189aa6dc13a1780c6dd7f6e559 not found: ID does not exist"
Jul 28 11:19:47.139253 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.139221    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"]
Jul 28 11:19:47.143205 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.143175    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-4cda06/rate-test-model-4cda06-kserve-686754c-jrcbt"]
Jul 28 11:19:47.495339 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:47.495302    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc524d3b-5bc1-49af-96e0-35060e9e1fc7" path="/var/lib/kubelet/pods/cc524d3b-5bc1-49af-96e0-35060e9e1fc7/volumes"
Jul 28 11:19:48.124932 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:48.124893    2578 generic.go:356] "Generic (PLEG): container finished" podID="e134db85-15f0-4354-bee2-5019e6778480" containerID="12a177f204438c182640b04742b936cd792e71506d98cd52b8a6a09300da831a" exitCode=0
Jul 28 11:19:48.125093 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:48.124984    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451111936" event={"ID":"e134db85-15f0-4354-bee2-5019e6778480","Type":"ContainerDied","Data":"12a177f204438c182640b04742b936cd792e71506d98cd52b8a6a09300da831a"}
Jul 28 11:19:49.268711 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.268678    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:49.288087 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.288049    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6975-140712451111936_e134db85-15f0-4354-bee2-5019e6778480/test-curl-6975-140712451111936/0.log"
Jul 28 11:19:49.311432 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.311384    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:49.314835 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.314791    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:49.402580 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.402481    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrrjx\" (UniqueName: \"kubernetes.io/projected/e134db85-15f0-4354-bee2-5019e6778480-kube-api-access-mrrjx\") pod \"e134db85-15f0-4354-bee2-5019e6778480\" (UID: \"e134db85-15f0-4354-bee2-5019e6778480\") "
Jul 28 11:19:49.404770 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.404746    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e134db85-15f0-4354-bee2-5019e6778480-kube-api-access-mrrjx" (OuterVolumeSpecName: "kube-api-access-mrrjx") pod "e134db85-15f0-4354-bee2-5019e6778480" (UID: "e134db85-15f0-4354-bee2-5019e6778480"). InnerVolumeSpecName "kube-api-access-mrrjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:19:49.462779 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.462740    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:49.463104 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.463090    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e134db85-15f0-4354-bee2-5019e6778480" containerName="test-curl-6975-140712451111936"
Jul 28 11:19:49.463172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.463105    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e134db85-15f0-4354-bee2-5019e6778480" containerName="test-curl-6975-140712451111936"
Jul 28 11:19:49.463172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.463128    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc524d3b-5bc1-49af-96e0-35060e9e1fc7" containerName="main"
Jul 28 11:19:49.463172 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.463133    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc524d3b-5bc1-49af-96e0-35060e9e1fc7" containerName="main"
Jul 28 11:19:49.463348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.463179    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc524d3b-5bc1-49af-96e0-35060e9e1fc7" containerName="main"
Jul 28 11:19:49.463348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.463187    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e134db85-15f0-4354-bee2-5019e6778480" containerName="test-curl-6975-140712451111936"
Jul 28 11:19:49.466190 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.466169    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:49.470711 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.470684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:49.495529 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.495491    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e134db85-15f0-4354-bee2-5019e6778480" path="/var/lib/kubelet/pods/e134db85-15f0-4354-bee2-5019e6778480/volumes"
Jul 28 11:19:49.503160 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.503127    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrjx\" (UniqueName: \"kubernetes.io/projected/e134db85-15f0-4354-bee2-5019e6778480-kube-api-access-mrrjx\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:49.604324 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.604247    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plf9z\" (UniqueName: \"kubernetes.io/projected/8de94148-0085-4921-8455-2736a5f327d7-kube-api-access-plf9z\") pod \"test-curl-6975-140712451111936\" (UID: \"8de94148-0085-4921-8455-2736a5f327d7\") " pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:49.705623 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.705589    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plf9z\" (UniqueName: \"kubernetes.io/projected/8de94148-0085-4921-8455-2736a5f327d7-kube-api-access-plf9z\") pod \"test-curl-6975-140712451111936\" (UID: \"8de94148-0085-4921-8455-2736a5f327d7\") " pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:49.713710 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.713673    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plf9z\" (UniqueName: \"kubernetes.io/projected/8de94148-0085-4921-8455-2736a5f327d7-kube-api-access-plf9z\") pod \"test-curl-6975-140712451111936\" (UID: \"8de94148-0085-4921-8455-2736a5f327d7\") " pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:49.777290 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:49.777244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:50.119037 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:50.119007    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:50.121366 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:19:50.121337    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8de94148_0085_4921_8455_2736a5f327d7.slice/crio-5f15f2c9a6b8a11a32217f138983301d5910bbe498780bccb987b3d990355e59 WatchSource:0}: Error finding container 5f15f2c9a6b8a11a32217f138983301d5910bbe498780bccb987b3d990355e59: Status 404 returned error can't find the container with id 5f15f2c9a6b8a11a32217f138983301d5910bbe498780bccb987b3d990355e59
Jul 28 11:19:50.136050 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:50.136013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451111936" event={"ID":"8de94148-0085-4921-8455-2736a5f327d7","Type":"ContainerStarted","Data":"5f15f2c9a6b8a11a32217f138983301d5910bbe498780bccb987b3d990355e59"}
Jul 28 11:19:50.137304 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:50.137239    2578 scope.go:117] "RemoveContainer" containerID="12a177f204438c182640b04742b936cd792e71506d98cd52b8a6a09300da831a"
Jul 28 11:19:50.137304 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:50.137250    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:51.145064 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:51.145020    2578 generic.go:356] "Generic (PLEG): container finished" podID="8de94148-0085-4921-8455-2736a5f327d7" containerID="0ddcf40080a1b82a633cc5543f1e83edd6bb99663dd873c04ec16cbe8b55046e" exitCode=0
Jul 28 11:19:51.145536 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:51.145106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-6975-140712451111936" event={"ID":"8de94148-0085-4921-8455-2736a5f327d7","Type":"ContainerDied","Data":"0ddcf40080a1b82a633cc5543f1e83edd6bb99663dd873c04ec16cbe8b55046e"}
Jul 28 11:19:52.290856 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.290829    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:52.310467 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.310417    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-6975-140712451111936_8de94148-0085-4921-8455-2736a5f327d7/test-curl-6975-140712451111936/0.log"
Jul 28 11:19:52.336114 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.336075    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:52.339443 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.339412    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-6975-140712451111936"]
Jul 28 11:19:52.431047 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.431010    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-plf9z\" (UniqueName: \"kubernetes.io/projected/8de94148-0085-4921-8455-2736a5f327d7-kube-api-access-plf9z\") pod \"8de94148-0085-4921-8455-2736a5f327d7\" (UID: \"8de94148-0085-4921-8455-2736a5f327d7\") "
Jul 28 11:19:52.433410 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.433375    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8de94148-0085-4921-8455-2736a5f327d7-kube-api-access-plf9z" (OuterVolumeSpecName: "kube-api-access-plf9z") pod "8de94148-0085-4921-8455-2736a5f327d7" (UID: "8de94148-0085-4921-8455-2736a5f327d7"). InnerVolumeSpecName "kube-api-access-plf9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:19:52.532142 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:52.532101    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plf9z\" (UniqueName: \"kubernetes.io/projected/8de94148-0085-4921-8455-2736a5f327d7-kube-api-access-plf9z\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:19:53.157189 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.157161    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:53.157387 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.157205    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f15f2c9a6b8a11a32217f138983301d5910bbe498780bccb987b3d990355e59"
Jul 28 11:19:53.167194 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:19:53.167162    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6975-140712451111936\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="8de94148-0085-4921-8455-2736a5f327d7" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:53.286938 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.286893    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"]
Jul 28 11:19:53.287357 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.287339    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8de94148-0085-4921-8455-2736a5f327d7" containerName="test-curl-6975-140712451111936"
Jul 28 11:19:53.287459 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.287359    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8de94148-0085-4921-8455-2736a5f327d7" containerName="test-curl-6975-140712451111936"
Jul 28 11:19:53.287519 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.287457    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8de94148-0085-4921-8455-2736a5f327d7" containerName="test-curl-6975-140712451111936"
Jul 28 11:19:53.290582 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.290559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.293077 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.293047    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1c9694-openshift-default-dockercfg-lxmtt\""
Jul 28 11:19:53.301798 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.301771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"]
Jul 28 11:19:53.320549 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:19:53.320507    2578 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-6975-140712451111936\" is forbidden: User \"system:node:ip-10-0-131-67.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-131-67.ec2.internal' and this object" podUID="8de94148-0085-4921-8455-2736a5f327d7" pod="opendatahub/test-curl-6975-140712451111936"
Jul 28 11:19:53.439499 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439454    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-credential-socket\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439499 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-data\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439512    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-podinfo\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-certs\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92k7\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-kube-api-access-m92k7\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439683    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-envoy\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439844 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439729    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-token\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439844 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istiod-ca-cert\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.439844 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.439791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-socket\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.494412 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.494378    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8de94148-0085-4921-8455-2736a5f327d7" path="/var/lib/kubelet/pods/8de94148-0085-4921-8455-2736a5f327d7/volumes"
Jul 28 11:19:53.540313 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-credential-socket\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540322    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-data\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-podinfo\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540376    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-certs\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92k7\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-kube-api-access-m92k7\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540490 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-envoy\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540500    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-token\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istiod-ca-cert\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-socket\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540730    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-credential-socket\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.540965 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-data\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.541026 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.540966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-certs\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.541078 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.541035    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-socket\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.541311 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.541249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istiod-ca-cert\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.542983 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.542957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-envoy\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.543079 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.543069    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-podinfo\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.552013 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.551979    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-token\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.552326 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.552307    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92k7\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-kube-api-access-m92k7\") pod \"e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") " pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.602658 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.602622    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:53.742443 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.742408    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"]
Jul 28 11:19:53.744258 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:19:53.744231    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2f26d9_11ad_46da_8982_0f2d2193a4e4.slice/crio-32ea8bf2e4929423b581e89b6c5f4577754f2fc5973ac35af3e7f0057d2eb4fb WatchSource:0}: Error finding container 32ea8bf2e4929423b581e89b6c5f4577754f2fc5973ac35af3e7f0057d2eb4fb: Status 404 returned error can't find the container with id 32ea8bf2e4929423b581e89b6c5f4577754f2fc5973ac35af3e7f0057d2eb4fb
Jul 28 11:19:53.746381 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.746351    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:19:53.746449 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.746419    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:19:53.746489 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:53.746447    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:19:54.165315 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:54.165201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" event={"ID":"2d2f26d9-11ad-46da-8982-0f2d2193a4e4","Type":"ContainerStarted","Data":"baa36c5844b9e6d705420b1f77e3e0021572879a9c1a35bc4bf63b9202387d43"}
Jul 28 11:19:54.165315 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:54.165264    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" event={"ID":"2d2f26d9-11ad-46da-8982-0f2d2193a4e4","Type":"ContainerStarted","Data":"32ea8bf2e4929423b581e89b6c5f4577754f2fc5973ac35af3e7f0057d2eb4fb"}
Jul 28 11:19:54.187541 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:54.187484    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podStartSLOduration=1.187465017 podStartE2EDuration="1.187465017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:19:54.185195815 +0000 UTC m=+3385.227287968" watchObservedRunningTime="2026-07-28 11:19:54.187465017 +0000 UTC m=+3385.229557169"
Jul 28 11:19:54.603465 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:54.603425    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:54.608677 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:54.608650    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:55.170611 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:55.170570    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:19:55.171692 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:19:55.171666    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:20:11.212473 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:11.212421    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 11:20:11.212904 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:11.212512    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 11:20:12.270468 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:12.270433    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerID="baa36c5844b9e6d705420b1f77e3e0021572879a9c1a35bc4bf63b9202387d43" exitCode=0
Jul 28 11:20:12.271024 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:12.270511    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" event={"ID":"2d2f26d9-11ad-46da-8982-0f2d2193a4e4","Type":"ContainerDied","Data":"baa36c5844b9e6d705420b1f77e3e0021572879a9c1a35bc4bf63b9202387d43"}
Jul 28 11:20:12.271093 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:12.271040    2578 scope.go:117] "RemoveContainer" containerID="baa36c5844b9e6d705420b1f77e3e0021572879a9c1a35bc4bf63b9202387d43"
Jul 28 11:20:12.272166 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:12.272139    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:20:12.272232 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:12.272209    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:20:12.272289 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:12.272241    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 11:20:13.277857 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:13.277820    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" event={"ID":"2d2f26d9-11ad-46da-8982-0f2d2193a4e4","Type":"ContainerStarted","Data":"9864ea6d87216d387b286415ebcaa18b8fe57a2d0c1e6cb7df84fd34dd50d30d"}
Jul 28 11:20:13.603065 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:13.602976    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:20:13.603337 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:13.603259    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Jul 28 11:20:13.603501 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:13.603398    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Jul 28 11:20:14.603407 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:14.603371    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Jul 28 11:20:14.603992 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:14.603959    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Jul 28 11:20:15.603472 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.603431    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Jul 28 11:20:15.604082 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.604046    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Jul 28 11:20:15.948832 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.948780    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"]
Jul 28 11:20:15.952130 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.952107    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:15.954818 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.954793    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a93417\""
Jul 28 11:20:15.955062 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.955040    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a93417-dockercfg-7drf4\""
Jul 28 11:20:15.963861 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:15.963829    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"]
Jul 28 11:20:16.056014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.055977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjvzf\" (UniqueName: \"kubernetes.io/projected/ee712416-5388-43c7-bd79-11838165f5f2-kube-api-access-bjvzf\") pod \"payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.056014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.056025    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee712416-5388-43c7-bd79-11838165f5f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.157191 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.157151    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjvzf\" (UniqueName: \"kubernetes.io/projected/ee712416-5388-43c7-bd79-11838165f5f2-kube-api-access-bjvzf\") pod \"payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.157191 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.157198    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee712416-5388-43c7-bd79-11838165f5f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.157933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.157914    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee712416-5388-43c7-bd79-11838165f5f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.167413 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.167388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjvzf\" (UniqueName: \"kubernetes.io/projected/ee712416-5388-43c7-bd79-11838165f5f2-kube-api-access-bjvzf\") pod \"payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.264419 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.264318    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:16.453682 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.453655    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"]
Jul 28 11:20:16.456323 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:20:16.456257    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee712416_5388_43c7_bd79_11838165f5f2.slice/crio-f938916e5a346b405babda46d3688b52f3c333537f5545b1063cc4fde3544c08 WatchSource:0}: Error finding container f938916e5a346b405babda46d3688b52f3c333537f5545b1063cc4fde3544c08: Status 404 returned error can't find the container with id f938916e5a346b405babda46d3688b52f3c333537f5545b1063cc4fde3544c08
Jul 28 11:20:16.603831 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.603354    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Jul 28 11:20:16.603831 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:16.603428    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Jul 28 11:20:17.300916 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.300873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm" event={"ID":"ee712416-5388-43c7-bd79-11838165f5f2","Type":"ContainerStarted","Data":"e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f"}
Jul 28 11:20:17.301127 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.300936    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm" event={"ID":"ee712416-5388-43c7-bd79-11838165f5f2","Type":"ContainerStarted","Data":"f938916e5a346b405babda46d3688b52f3c333537f5545b1063cc4fde3544c08"}
Jul 28 11:20:17.301127 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.300978    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:17.320914 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.320851    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm" podStartSLOduration=2.320833026 podStartE2EDuration="2.320833026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:20:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:20:17.318112157 +0000 UTC m=+3408.360204323" watchObservedRunningTime="2026-07-28 11:20:17.320833026 +0000 UTC m=+3408.362925178"
Jul 28 11:20:17.607560 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.607472    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:20:17.608006 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.607986    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:20:17.608756 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:17.608737    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:20:28.309627 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:28.309545    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:20:37.440164 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:37.440134    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm_ee712416-5388-43c7-bd79-11838165f5f2/payload-processing/0.log"
Jul 28 11:20:48.830463 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.830420    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"]
Jul 28 11:20:48.834904 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.834875    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.838532 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.838496    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1c9694\"/\"kube-root-ca.crt\""
Jul 28 11:20:48.838532 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.838496    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1c9694\"/\"ipp-route-1c9694-kserve-self-signed-certs\""
Jul 28 11:20:48.838908 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.838510    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-1c9694\"/\"openshift-service-ca.crt\""
Jul 28 11:20:48.838908 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.838511    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-1c9694\"/\"default-dockercfg-wxq2t\""
Jul 28 11:20:48.845991 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.845959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"]
Jul 28 11:20:48.856533 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.856490    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-tmp-dir\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.856718 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.856557    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-dshm\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.856718 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.856591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-model-cache\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.856718 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.856614    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877fe849-cd6b-456a-9ae0-1a957fc406cd-tls-certs\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.856851 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.856719    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5qtg\" (UniqueName: \"kubernetes.io/projected/877fe849-cd6b-456a-9ae0-1a957fc406cd-kube-api-access-j5qtg\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.856851 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.856774    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-home\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5qtg\" (UniqueName: \"kubernetes.io/projected/877fe849-cd6b-456a-9ae0-1a957fc406cd-kube-api-access-j5qtg\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958233 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-home\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958548 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958320    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-tmp-dir\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958548 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958350    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-dshm\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958548 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-model-cache\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958548 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877fe849-cd6b-456a-9ae0-1a957fc406cd-tls-certs\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958769 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958749    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-tmp-dir\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958811 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958797    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-home\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.958882 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.958864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-model-cache\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.961023 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.961001    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-dshm\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.961341 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.961322    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877fe849-cd6b-456a-9ae0-1a957fc406cd-tls-certs\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:48.966285 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:48.966240    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5qtg\" (UniqueName: \"kubernetes.io/projected/877fe849-cd6b-456a-9ae0-1a957fc406cd-kube-api-access-j5qtg\") pod \"ipp-route-1c9694-kserve-85c9c67755-pz7gj\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") " pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:49.148348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:49.148231    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:49.291210 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:49.291185    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"]
Jul 28 11:20:49.294004 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:20:49.293976    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod877fe849_cd6b_456a_9ae0_1a957fc406cd.slice/crio-41d427a74b8a6968e87b056f7a02b64ae7a342c73a5e14bbbaaa3873f8a0b6ba WatchSource:0}: Error finding container 41d427a74b8a6968e87b056f7a02b64ae7a342c73a5e14bbbaaa3873f8a0b6ba: Status 404 returned error can't find the container with id 41d427a74b8a6968e87b056f7a02b64ae7a342c73a5e14bbbaaa3873f8a0b6ba
Jul 28 11:20:49.464744 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:49.464709    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj" event={"ID":"877fe849-cd6b-456a-9ae0-1a957fc406cd","Type":"ContainerStarted","Data":"a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be"}
Jul 28 11:20:49.464957 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:49.464756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj" event={"ID":"877fe849-cd6b-456a-9ae0-1a957fc406cd","Type":"ContainerStarted","Data":"41d427a74b8a6968e87b056f7a02b64ae7a342c73a5e14bbbaaa3873f8a0b6ba"}
Jul 28 11:20:49.481899 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:49.481841    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj" podStartSLOduration=1.4818222300000001 podStartE2EDuration="1.48182223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:20:49.480203472 +0000 UTC m=+3440.522295625" watchObservedRunningTime="2026-07-28 11:20:49.48182223 +0000 UTC m=+3440.523914383"
Jul 28 11:20:59.148684 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:59.148636    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:59.149155 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:59.148699    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:59.161552 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:59.161516    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:20:59.524736 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:20:59.524700    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:21:35.247984 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:21:35.247955    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm_ee712416-5388-43c7-bd79-11838165f5f2/payload-processing/0.log"
Jul 28 11:22:48.548750 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:22:48.548705    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"]
Jul 28 11:23:03.568892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.568828    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj" podUID="877fe849-cd6b-456a-9ae0-1a957fc406cd" containerName="main" containerID="cri-o://a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be" gracePeriod=45
Jul 28 11:23:03.828999 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.828908    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:23:03.913615 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-tmp-dir\") pod \"877fe849-cd6b-456a-9ae0-1a957fc406cd\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") "
Jul 28 11:23:03.913615 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913616    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-model-cache\") pod \"877fe849-cd6b-456a-9ae0-1a957fc406cd\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") "
Jul 28 11:23:03.913908 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913662    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877fe849-cd6b-456a-9ae0-1a957fc406cd-tls-certs\") pod \"877fe849-cd6b-456a-9ae0-1a957fc406cd\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") "
Jul 28 11:23:03.913908 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913690    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-home\") pod \"877fe849-cd6b-456a-9ae0-1a957fc406cd\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") "
Jul 28 11:23:03.913908 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913712    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5qtg\" (UniqueName: \"kubernetes.io/projected/877fe849-cd6b-456a-9ae0-1a957fc406cd-kube-api-access-j5qtg\") pod \"877fe849-cd6b-456a-9ae0-1a957fc406cd\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") "
Jul 28 11:23:03.913908 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-dshm\") pod \"877fe849-cd6b-456a-9ae0-1a957fc406cd\" (UID: \"877fe849-cd6b-456a-9ae0-1a957fc406cd\") "
Jul 28 11:23:03.914527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.913921    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "877fe849-cd6b-456a-9ae0-1a957fc406cd" (UID: "877fe849-cd6b-456a-9ae0-1a957fc406cd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:03.914527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.914142    2578 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-tmp-dir\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:03.914527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.914210    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-model-cache" (OuterVolumeSpecName: "model-cache") pod "877fe849-cd6b-456a-9ae0-1a957fc406cd" (UID: "877fe849-cd6b-456a-9ae0-1a957fc406cd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:03.914527 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.914403    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-home" (OuterVolumeSpecName: "home") pod "877fe849-cd6b-456a-9ae0-1a957fc406cd" (UID: "877fe849-cd6b-456a-9ae0-1a957fc406cd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:03.916477 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.916438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/877fe849-cd6b-456a-9ae0-1a957fc406cd-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "877fe849-cd6b-456a-9ae0-1a957fc406cd" (UID: "877fe849-cd6b-456a-9ae0-1a957fc406cd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 11:23:03.916614 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.916475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-dshm" (OuterVolumeSpecName: "dshm") pod "877fe849-cd6b-456a-9ae0-1a957fc406cd" (UID: "877fe849-cd6b-456a-9ae0-1a957fc406cd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:03.916614 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:03.916604    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877fe849-cd6b-456a-9ae0-1a957fc406cd-kube-api-access-j5qtg" (OuterVolumeSpecName: "kube-api-access-j5qtg") pod "877fe849-cd6b-456a-9ae0-1a957fc406cd" (UID: "877fe849-cd6b-456a-9ae0-1a957fc406cd"). InnerVolumeSpecName "kube-api-access-j5qtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:23:04.015379 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.015335    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5qtg\" (UniqueName: \"kubernetes.io/projected/877fe849-cd6b-456a-9ae0-1a957fc406cd-kube-api-access-j5qtg\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:04.015379 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.015371    2578 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-dshm\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:04.015379 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.015382    2578 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-model-cache\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:04.015379 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.015390    2578 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/877fe849-cd6b-456a-9ae0-1a957fc406cd-tls-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:04.015669 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.015398    2578 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/877fe849-cd6b-456a-9ae0-1a957fc406cd-home\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:04.135897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.135791    2578 generic.go:356] "Generic (PLEG): container finished" podID="877fe849-cd6b-456a-9ae0-1a957fc406cd" containerID="a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be" exitCode=0
Jul 28 11:23:04.135897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.135840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"
Jul 28 11:23:04.135897 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.135879    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj" event={"ID":"877fe849-cd6b-456a-9ae0-1a957fc406cd","Type":"ContainerDied","Data":"a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be"}
Jul 28 11:23:04.136197 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.135932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj" event={"ID":"877fe849-cd6b-456a-9ae0-1a957fc406cd","Type":"ContainerDied","Data":"41d427a74b8a6968e87b056f7a02b64ae7a342c73a5e14bbbaaa3873f8a0b6ba"}
Jul 28 11:23:04.136197 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.135968    2578 scope.go:117] "RemoveContainer" containerID="a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be"
Jul 28 11:23:04.146029 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.146009    2578 scope.go:117] "RemoveContainer" containerID="a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be"
Jul 28 11:23:04.146378 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:23:04.146359    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be\": container with ID starting with a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be not found: ID does not exist" containerID="a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be"
Jul 28 11:23:04.146454 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.146386    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be"} err="failed to get container status \"a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be\": rpc error: code = NotFound desc = could not find container \"a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be\": container with ID starting with a9cf711db37f6723955471de096f6a3f41136145d455bef770447b24e8f2a9be not found: ID does not exist"
Jul 28 11:23:04.171175 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.171137    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"]
Jul 28 11:23:04.180458 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:04.180419    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-1c9694/ipp-route-1c9694-kserve-85c9c67755-pz7gj"]
Jul 28 11:23:05.494836 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:05.494789    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877fe849-cd6b-456a-9ae0-1a957fc406cd" path="/var/lib/kubelet/pods/877fe849-cd6b-456a-9ae0-1a957fc406cd/volumes"
Jul 28 11:23:30.095589 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:30.095559    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:23:30.102116 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:30.102090    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jmrkn_e3c17446-0ae7-4e7f-bbf6-669b2625cbed/ovn-acl-logging/0.log"
Jul 28 11:23:33.106831 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:33.106787    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"]
Jul 28 11:23:33.107350 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:33.107059    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy" containerID="cri-o://9864ea6d87216d387b286415ebcaa18b8fe57a2d0c1e6cb7df84fd34dd50d30d" gracePeriod=30
Jul 28 11:23:38.309068 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.309032    2578 generic.go:356] "Generic (PLEG): container finished" podID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerID="9864ea6d87216d387b286415ebcaa18b8fe57a2d0c1e6cb7df84fd34dd50d30d" exitCode=0
Jul 28 11:23:38.309596 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.309099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" event={"ID":"2d2f26d9-11ad-46da-8982-0f2d2193a4e4","Type":"ContainerDied","Data":"9864ea6d87216d387b286415ebcaa18b8fe57a2d0c1e6cb7df84fd34dd50d30d"}
Jul 28 11:23:38.309596 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.309184    2578 scope.go:117] "RemoveContainer" containerID="baa36c5844b9e6d705420b1f77e3e0021572879a9c1a35bc4bf63b9202387d43"
Jul 28 11:23:38.358607 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.358574    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:23:38.527998 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.527954    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-podinfo\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.527998 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528013    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-envoy\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-data\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528078    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-credential-socket\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528096    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-token\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528110    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istiod-ca-cert\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528132    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-certs\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m92k7\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-kube-api-access-m92k7\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528310 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528240    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-socket\") pod \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\" (UID: \"2d2f26d9-11ad-46da-8982-0f2d2193a4e4\") "
Jul 28 11:23:38.528675 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:38.528675 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528570    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.528675 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528554    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:38.528675 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528638    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:38.529110 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.528738    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:23:38.529336 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.529306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-data" (OuterVolumeSpecName: "istio-data") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:38.530520 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.530489    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:23:38.530520 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.530493    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-token" (OuterVolumeSpecName: "istio-token") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:23:38.530674 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.530575    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:23:38.530753 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.530734    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-kube-api-access-m92k7" (OuterVolumeSpecName: "kube-api-access-m92k7") pod "2d2f26d9-11ad-46da-8982-0f2d2193a4e4" (UID: "2d2f26d9-11ad-46da-8982-0f2d2193a4e4"). InnerVolumeSpecName "kube-api-access-m92k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:23:38.629367 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629326    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629367 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629357    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629367 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629366    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629367 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629375    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m92k7\" (UniqueName: \"kubernetes.io/projected/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-kube-api-access-m92k7\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629642 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629383    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629642 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629392    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629642 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629402    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:38.629642 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:38.629409    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d2f26d9-11ad-46da-8982-0f2d2193a4e4-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:39.316115 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:39.316081    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"
Jul 28 11:23:39.316115 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:39.316093    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w" event={"ID":"2d2f26d9-11ad-46da-8982-0f2d2193a4e4","Type":"ContainerDied","Data":"32ea8bf2e4929423b581e89b6c5f4577754f2fc5973ac35af3e7f0057d2eb4fb"}
Jul 28 11:23:39.316618 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:39.316148    2578 scope.go:117] "RemoveContainer" containerID="9864ea6d87216d387b286415ebcaa18b8fe57a2d0c1e6cb7df84fd34dd50d30d"
Jul 28 11:23:39.338632 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:39.338592    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"]
Jul 28 11:23:39.346685 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:39.346656    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1c9694-openshift-default-6f4b57d458-z6r6w"]
Jul 28 11:23:39.497109 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:39.497065    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" path="/var/lib/kubelet/pods/2d2f26d9-11ad-46da-8982-0f2d2193a4e4/volumes"
Jul 28 11:23:45.322946 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.322910    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"]
Jul 28 11:23:45.323457 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.323130    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm" podUID="ee712416-5388-43c7-bd79-11838165f5f2" containerName="payload-processing" containerID="cri-o://e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f" gracePeriod=30
Jul 28 11:23:45.575663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.575587    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:23:45.693694 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.693646    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjvzf\" (UniqueName: \"kubernetes.io/projected/ee712416-5388-43c7-bd79-11838165f5f2-kube-api-access-bjvzf\") pod \"ee712416-5388-43c7-bd79-11838165f5f2\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") "
Jul 28 11:23:45.693873 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.693751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee712416-5388-43c7-bd79-11838165f5f2-plugins-config-volume\") pod \"ee712416-5388-43c7-bd79-11838165f5f2\" (UID: \"ee712416-5388-43c7-bd79-11838165f5f2\") "
Jul 28 11:23:45.694162 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.694136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee712416-5388-43c7-bd79-11838165f5f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee712416-5388-43c7-bd79-11838165f5f2" (UID: "ee712416-5388-43c7-bd79-11838165f5f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:23:45.696037 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.696005    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee712416-5388-43c7-bd79-11838165f5f2-kube-api-access-bjvzf" (OuterVolumeSpecName: "kube-api-access-bjvzf") pod "ee712416-5388-43c7-bd79-11838165f5f2" (UID: "ee712416-5388-43c7-bd79-11838165f5f2"). InnerVolumeSpecName "kube-api-access-bjvzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:23:45.795237 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.795195    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee712416-5388-43c7-bd79-11838165f5f2-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:45.795237 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:45.795227    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjvzf\" (UniqueName: \"kubernetes.io/projected/ee712416-5388-43c7-bd79-11838165f5f2-kube-api-access-bjvzf\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:23:46.351698 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.351665    2578 generic.go:356] "Generic (PLEG): container finished" podID="ee712416-5388-43c7-bd79-11838165f5f2" containerID="e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f" exitCode=0
Jul 28 11:23:46.352130 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.351713    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"
Jul 28 11:23:46.352130 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.351764    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm" event={"ID":"ee712416-5388-43c7-bd79-11838165f5f2","Type":"ContainerDied","Data":"e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f"}
Jul 28 11:23:46.352130 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.351831    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm" event={"ID":"ee712416-5388-43c7-bd79-11838165f5f2","Type":"ContainerDied","Data":"f938916e5a346b405babda46d3688b52f3c333537f5545b1063cc4fde3544c08"}
Jul 28 11:23:46.352130 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.351866    2578 scope.go:117] "RemoveContainer" containerID="e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f"
Jul 28 11:23:46.361593 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.361576    2578 scope.go:117] "RemoveContainer" containerID="e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f"
Jul 28 11:23:46.361869 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:23:46.361849    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f\": container with ID starting with e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f not found: ID does not exist" containerID="e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f"
Jul 28 11:23:46.361929 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.361883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f"} err="failed to get container status \"e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f\": rpc error: code = NotFound desc = could not find container \"e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f\": container with ID starting with e5c09a11b09dcd5842f8e5e58ad68978a0f1e7ec8ad6ab63901fdf7e1ce98f3f not found: ID does not exist"
Jul 28 11:23:46.375175 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.375144    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"]
Jul 28 11:23:46.381998 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:46.381967    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a93417-8487bd74b-m95jm"]
Jul 28 11:23:47.494891 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:47.494860    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee712416-5388-43c7-bd79-11838165f5f2" path="/var/lib/kubelet/pods/ee712416-5388-43c7-bd79-11838165f5f2/volumes"
Jul 28 11:23:56.491519 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491480    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"]
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491833    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877fe849-cd6b-456a-9ae0-1a957fc406cd" containerName="main"
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491844    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="877fe849-cd6b-456a-9ae0-1a957fc406cd" containerName="main"
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491860    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy"
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491866    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy"
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491883    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy"
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491888    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy"
Jul 28 11:23:56.491892 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491893    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee712416-5388-43c7-bd79-11838165f5f2" containerName="payload-processing"
Jul 28 11:23:56.492113 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491899    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee712416-5388-43c7-bd79-11838165f5f2" containerName="payload-processing"
Jul 28 11:23:56.492113 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491948    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee712416-5388-43c7-bd79-11838165f5f2" containerName="payload-processing"
Jul 28 11:23:56.492113 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491955    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy"
Jul 28 11:23:56.492113 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491962    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d2f26d9-11ad-46da-8982-0f2d2193a4e4" containerName="istio-proxy"
Jul 28 11:23:56.492113 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.491968    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="877fe849-cd6b-456a-9ae0-1a957fc406cd" containerName="main"
Jul 28 11:23:56.496375 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.496356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:23:56.498943 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.498920    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 11:23:56.499105 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.499092    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:23:56.504380 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.504344    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"]
Jul 28 11:23:56.582465 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.582428    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx\" (UID: \"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:23:56.683101 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.683061    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx\" (UID: \"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:23:56.683672 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.683652    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx\" (UID: \"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:23:56.807702 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.807614    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:23:56.969301 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:56.969254    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"]
Jul 28 11:23:56.972933 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:23:56.972901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ba6aeb_1bb4_4339_bf00_afd6e5ffb69a.slice/crio-31ccffb128a5f58f2ad589c35c7e45a8b55a83c77fa1c4b775f1fbd89faf9f3c WatchSource:0}: Error finding container 31ccffb128a5f58f2ad589c35c7e45a8b55a83c77fa1c4b775f1fbd89faf9f3c: Status 404 returned error can't find the container with id 31ccffb128a5f58f2ad589c35c7e45a8b55a83c77fa1c4b775f1fbd89faf9f3c
Jul 28 11:23:57.409366 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:57.409332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx" event={"ID":"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a","Type":"ContainerStarted","Data":"9a6eba19f22fb7fe5cff9711f03f7acff27d23edc3036a701d1675c208c3852a"}
Jul 28 11:23:57.409522 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:57.409386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx" event={"ID":"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a","Type":"ContainerStarted","Data":"31ccffb128a5f58f2ad589c35c7e45a8b55a83c77fa1c4b775f1fbd89faf9f3c"}
Jul 28 11:23:57.426736 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:57.426683    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx" podStartSLOduration=1.426668584 podStartE2EDuration="1.426668584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:23:57.425663878 +0000 UTC m=+3628.467756031" watchObservedRunningTime="2026-07-28 11:23:57.426668584 +0000 UTC m=+3628.468760737"
Jul 28 11:23:58.415578 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:58.415542    2578 generic.go:356] "Generic (PLEG): container finished" podID="72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a" containerID="9a6eba19f22fb7fe5cff9711f03f7acff27d23edc3036a701d1675c208c3852a" exitCode=0
Jul 28 11:23:58.415938 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:58.415638    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx" event={"ID":"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a","Type":"ContainerDied","Data":"9a6eba19f22fb7fe5cff9711f03f7acff27d23edc3036a701d1675c208c3852a"}
Jul 28 11:23:59.549890 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:59.549860    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:23:59.611202 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:59.611167    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a-maas-api-service-ca\") pod \"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a\" (UID: \"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a\") "
Jul 28 11:23:59.611558 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:59.611535    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a" (UID: "72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:23:59.712700 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:23:59.712613    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:00.427307 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:00.427262    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"
Jul 28 11:24:00.427307 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:00.427296    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx" event={"ID":"72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a","Type":"ContainerDied","Data":"31ccffb128a5f58f2ad589c35c7e45a8b55a83c77fa1c4b775f1fbd89faf9f3c"}
Jul 28 11:24:00.427534 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:00.427355    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31ccffb128a5f58f2ad589c35c7e45a8b55a83c77fa1c4b775f1fbd89faf9f3c"
Jul 28 11:24:07.813174 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:07.813138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"]
Jul 28 11:24:07.815829 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:07.815805    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4cda06-29753955-lmcwz"]
Jul 28 11:24:07.859979 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:07.859946    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"]
Jul 28 11:24:07.860171 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:07.860134    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r" podUID="c293f698-7836-4336-bcf1-b0a1fde79ebf" containerName="payload-processing" containerID="cri-o://e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6" gracePeriod=30
Jul 28 11:24:08.117006 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.116981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:24:08.189526 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.189485    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-94wnq\" (UniqueName: \"kubernetes.io/projected/c293f698-7836-4336-bcf1-b0a1fde79ebf-kube-api-access-94wnq\") pod \"c293f698-7836-4336-bcf1-b0a1fde79ebf\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") "
Jul 28 11:24:08.189728 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.189606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293f698-7836-4336-bcf1-b0a1fde79ebf-plugins-config-volume\") pod \"c293f698-7836-4336-bcf1-b0a1fde79ebf\" (UID: \"c293f698-7836-4336-bcf1-b0a1fde79ebf\") "
Jul 28 11:24:08.189925 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.189900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c293f698-7836-4336-bcf1-b0a1fde79ebf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c293f698-7836-4336-bcf1-b0a1fde79ebf" (UID: "c293f698-7836-4336-bcf1-b0a1fde79ebf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:08.191775 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.191747    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c293f698-7836-4336-bcf1-b0a1fde79ebf-kube-api-access-94wnq" (OuterVolumeSpecName: "kube-api-access-94wnq") pod "c293f698-7836-4336-bcf1-b0a1fde79ebf" (UID: "c293f698-7836-4336-bcf1-b0a1fde79ebf"). InnerVolumeSpecName "kube-api-access-94wnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:24:08.290518 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.290473    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293f698-7836-4336-bcf1-b0a1fde79ebf-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:08.290518 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.290511    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94wnq\" (UniqueName: \"kubernetes.io/projected/c293f698-7836-4336-bcf1-b0a1fde79ebf-kube-api-access-94wnq\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:08.467096 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.467057    2578 generic.go:356] "Generic (PLEG): container finished" podID="c293f698-7836-4336-bcf1-b0a1fde79ebf" containerID="e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6" exitCode=0
Jul 28 11:24:08.467283 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.467101    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"
Jul 28 11:24:08.467283 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.467141    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r" event={"ID":"c293f698-7836-4336-bcf1-b0a1fde79ebf","Type":"ContainerDied","Data":"e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6"}
Jul 28 11:24:08.467283 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.467204    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r" event={"ID":"c293f698-7836-4336-bcf1-b0a1fde79ebf","Type":"ContainerDied","Data":"18fe6b6b4c0006cf046b63dc500c23e68d160f59f2bc83dec76402c43bd06694"}
Jul 28 11:24:08.467283 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.467227    2578 scope.go:117] "RemoveContainer" containerID="e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6"
Jul 28 11:24:08.476663 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.476643    2578 scope.go:117] "RemoveContainer" containerID="e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6"
Jul 28 11:24:08.477044 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:24:08.477012    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6\": container with ID starting with e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6 not found: ID does not exist" containerID="e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6"
Jul 28 11:24:08.477110 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.477052    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6"} err="failed to get container status \"e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6\": rpc error: code = NotFound desc = could not find container \"e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6\": container with ID starting with e0517d79727f6b9e3424dafcf6b74e048b04a8961a58e021972d45f5e761f5c6 not found: ID does not exist"
Jul 28 11:24:08.488068 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.488043    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"]
Jul 28 11:24:08.492097 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:08.492076    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-4cda06-8b444bdfd-8kr7r"]
Jul 28 11:24:09.494140 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:09.494095    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b29d3b2-0957-46a6-a304-c085cbf41fb2" path="/var/lib/kubelet/pods/8b29d3b2-0957-46a6-a304-c085cbf41fb2/volumes"
Jul 28 11:24:09.494558 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:09.494450    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c293f698-7836-4336-bcf1-b0a1fde79ebf" path="/var/lib/kubelet/pods/c293f698-7836-4336-bcf1-b0a1fde79ebf/volumes"
Jul 28 11:24:11.580415 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:11.580376    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"]
Jul 28 11:24:11.584542 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:11.584504    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4cda06-68jnx"]
Jul 28 11:24:13.493770 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:13.493736    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a" path="/var/lib/kubelet/pods/72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a/volumes"
Jul 28 11:24:18.775807 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:18.775766    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"]
Jul 28 11:24:18.776222 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:18.776029    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy" containerID="cri-o://77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad" gracePeriod=30
Jul 28 11:24:19.263542 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263504    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"]
Jul 28 11:24:19.263890 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263876    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c293f698-7836-4336-bcf1-b0a1fde79ebf" containerName="payload-processing"
Jul 28 11:24:19.263958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263893    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c293f698-7836-4336-bcf1-b0a1fde79ebf" containerName="payload-processing"
Jul 28 11:24:19.263958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263903    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a" containerName="revoke-keys"
Jul 28 11:24:19.263958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263909    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a" containerName="revoke-keys"
Jul 28 11:24:19.264058 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263965    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ba6aeb-1bb4-4339-bf00-afd6e5ffb69a" containerName="revoke-keys"
Jul 28 11:24:19.264058 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.263976    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c293f698-7836-4336-bcf1-b0a1fde79ebf" containerName="payload-processing"
Jul 28 11:24:19.267250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.267227    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:19.271367 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.271338    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 11:24:19.271548 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.271337    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gvrfc\""
Jul 28 11:24:19.373627 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.373588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"]
Jul 28 11:24:19.391691 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.391656    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e313909-c157-4fe4-ba40-5966bf2e51f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf\" (UID: \"0e313909-c157-4fe4-ba40-5966bf2e51f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:19.492191 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.492155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e313909-c157-4fe4-ba40-5966bf2e51f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf\" (UID: \"0e313909-c157-4fe4-ba40-5966bf2e51f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:19.492832 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.492808    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e313909-c157-4fe4-ba40-5966bf2e51f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf\" (UID: \"0e313909-c157-4fe4-ba40-5966bf2e51f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:19.577818 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.577720    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:19.721733 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:19.721705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"]
Jul 28 11:24:19.723973 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:24:19.723945    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e313909_c157_4fe4_ba40_5966bf2e51f0.slice/crio-e72e4ab082f271cedaf618509671d3fa4283c73da090b7679cc7f7308d636470 WatchSource:0}: Error finding container e72e4ab082f271cedaf618509671d3fa4283c73da090b7679cc7f7308d636470: Status 404 returned error can't find the container with id e72e4ab082f271cedaf618509671d3fa4283c73da090b7679cc7f7308d636470
Jul 28 11:24:20.528870 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:20.528833    2578 generic.go:356] "Generic (PLEG): container finished" podID="0e313909-c157-4fe4-ba40-5966bf2e51f0" containerID="290e178b7696e1bb8479986b0f181fff680465e6d8c17c95dba7cd175b1faa27" exitCode=0
Jul 28 11:24:20.529248 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:20.528921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf" event={"ID":"0e313909-c157-4fe4-ba40-5966bf2e51f0","Type":"ContainerDied","Data":"290e178b7696e1bb8479986b0f181fff680465e6d8c17c95dba7cd175b1faa27"}
Jul 28 11:24:20.529248 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:20.528976    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf" event={"ID":"0e313909-c157-4fe4-ba40-5966bf2e51f0","Type":"ContainerStarted","Data":"e72e4ab082f271cedaf618509671d3fa4283c73da090b7679cc7f7308d636470"}
Jul 28 11:24:21.673496 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:21.673465    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:21.813795 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:21.813688    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e313909-c157-4fe4-ba40-5966bf2e51f0-maas-api-service-ca\") pod \"0e313909-c157-4fe4-ba40-5966bf2e51f0\" (UID: \"0e313909-c157-4fe4-ba40-5966bf2e51f0\") "
Jul 28 11:24:21.814079 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:21.814051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e313909-c157-4fe4-ba40-5966bf2e51f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e313909-c157-4fe4-ba40-5966bf2e51f0" (UID: "0e313909-c157-4fe4-ba40-5966bf2e51f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:21.914652 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:21.914614    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e313909-c157-4fe4-ba40-5966bf2e51f0-maas-api-service-ca\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:22.539952 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:22.539913    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"
Jul 28 11:24:22.540144 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:22.539912    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf" event={"ID":"0e313909-c157-4fe4-ba40-5966bf2e51f0","Type":"ContainerDied","Data":"e72e4ab082f271cedaf618509671d3fa4283c73da090b7679cc7f7308d636470"}
Jul 28 11:24:22.540144 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:22.540048    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e72e4ab082f271cedaf618509671d3fa4283c73da090b7679cc7f7308d636470"
Jul 28 11:24:24.040719 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.040627    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:24:24.134740 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134693    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kz2m\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-kube-api-access-2kz2m\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.134919 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134788    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istiod-ca-cert\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.134919 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134816    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-podinfo\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.134919 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134871    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-token\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.134919 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-credential-socket\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.135119 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134919    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-certs\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.135119 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134950    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-envoy\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.135119 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.134979    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-socket\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.135119 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.135034    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-data\") pod \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\" (UID: \"b130dafc-5b3d-4a62-bc3d-27087f3d3535\") "
Jul 28 11:24:24.135380 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.135234    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:24.135380 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.135343    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:24:24.135486 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.135402    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istiod-ca-cert\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.135547 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.135495    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:24:24.135599 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.135584    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:24:24.136484 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.136459    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-data" (OuterVolumeSpecName: "istio-data") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:24:24.137442 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.137417    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-token" (OuterVolumeSpecName: "istio-token") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:24:24.137548 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.137439    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-kube-api-access-2kz2m" (OuterVolumeSpecName: "kube-api-access-2kz2m") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "kube-api-access-2kz2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:24:24.137789 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.137771    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 11:24:24.137844 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.137781    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b130dafc-5b3d-4a62-bc3d-27087f3d3535" (UID: "b130dafc-5b3d-4a62-bc3d-27087f3d3535"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 11:24:24.236654 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236608    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-data\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236654 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236648    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kz2m\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-kube-api-access-2kz2m\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236654 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236659    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-podinfo\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236654 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236667    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-token\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236675    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-credential-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236683    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-certs\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236691    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-istio-envoy\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.236933 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.236698    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b130dafc-5b3d-4a62-bc3d-27087f3d3535-workload-socket\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:24.552913 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.552880    2578 generic.go:356] "Generic (PLEG): container finished" podID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerID="77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad" exitCode=0
Jul 28 11:24:24.553100 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.552936    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"
Jul 28 11:24:24.553100 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.552961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" event={"ID":"b130dafc-5b3d-4a62-bc3d-27087f3d3535","Type":"ContainerDied","Data":"77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad"}
Jul 28 11:24:24.553100 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.553014    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44" event={"ID":"b130dafc-5b3d-4a62-bc3d-27087f3d3535","Type":"ContainerDied","Data":"2a7d05ecbee695f8046180c8692e3f12164fac4d5e36a917e13019f0b915122b"}
Jul 28 11:24:24.553100 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.553038    2578 scope.go:117] "RemoveContainer" containerID="77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad"
Jul 28 11:24:24.562535 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.562516    2578 scope.go:117] "RemoveContainer" containerID="d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"
Jul 28 11:24:24.570898 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.570877    2578 scope.go:117] "RemoveContainer" containerID="77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad"
Jul 28 11:24:24.571214 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:24:24.571193    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad\": container with ID starting with 77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad not found: ID does not exist" containerID="77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad"
Jul 28 11:24:24.571260 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.571224    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad"} err="failed to get container status \"77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad\": rpc error: code = NotFound desc = could not find container \"77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad\": container with ID starting with 77c73ebf3487713fced2d36fcea2f36ea7f0600afc6e7d34cd265b88350eabad not found: ID does not exist"
Jul 28 11:24:24.571260 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.571243    2578 scope.go:117] "RemoveContainer" containerID="d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"
Jul 28 11:24:24.571539 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:24:24.571522    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08\": container with ID starting with d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08 not found: ID does not exist" containerID="d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"
Jul 28 11:24:24.571576 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.571546    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08"} err="failed to get container status \"d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08\": rpc error: code = NotFound desc = could not find container \"d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08\": container with ID starting with d475143f1fdf0a55b4b5ccb0dde8722aaf17e56f413d4ff2adc378b30aeace08 not found: ID does not exist"
Jul 28 11:24:24.576942 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.576911    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"]
Jul 28 11:24:24.583629 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:24.583599    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-4cda06-openshift-default-66c4bc98c6-tbg44"]
Jul 28 11:24:25.495747 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:25.495709    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" path="/var/lib/kubelet/pods/b130dafc-5b3d-4a62-bc3d-27087f3d3535/volumes"
Jul 28 11:24:30.339290 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:30.339234    2578 scope.go:117] "RemoveContainer" containerID="ca57b1d0d4dab17a3c09f6eb7423a49dba5582ad45ce2f2e76ee165762892f32"
Jul 28 11:24:30.998708 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:30.998669    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"]
Jul 28 11:24:30.998958 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:30.998926    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6" podUID="b35a494e-db46-4d09-a16b-b2883577e13b" containerName="payload-processing" containerID="cri-o://b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c" gracePeriod=30
Jul 28 11:24:31.009770 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.009737    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"]
Jul 28 11:24:31.009985 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.009936    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph" podUID="03a7bd96-1a17-4490-8ea8-b0697ac08239" containerName="payload-pre-processing" containerID="cri-o://6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444" gracePeriod=30
Jul 28 11:24:31.288767 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.288730    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:24:31.292471 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.292451    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:24:31.404333 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.404260    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35a494e-db46-4d09-a16b-b2883577e13b-plugins-config-volume\") pod \"b35a494e-db46-4d09-a16b-b2883577e13b\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") "
Jul 28 11:24:31.404782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.404350    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a7bd96-1a17-4490-8ea8-b0697ac08239-plugins-config-volume\") pod \"03a7bd96-1a17-4490-8ea8-b0697ac08239\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") "
Jul 28 11:24:31.404782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.404403    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwwhw\" (UniqueName: \"kubernetes.io/projected/b35a494e-db46-4d09-a16b-b2883577e13b-kube-api-access-nwwhw\") pod \"b35a494e-db46-4d09-a16b-b2883577e13b\" (UID: \"b35a494e-db46-4d09-a16b-b2883577e13b\") "
Jul 28 11:24:31.404782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.404434    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2zkv\" (UniqueName: \"kubernetes.io/projected/03a7bd96-1a17-4490-8ea8-b0697ac08239-kube-api-access-q2zkv\") pod \"03a7bd96-1a17-4490-8ea8-b0697ac08239\" (UID: \"03a7bd96-1a17-4490-8ea8-b0697ac08239\") "
Jul 28 11:24:31.404782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.404722    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a7bd96-1a17-4490-8ea8-b0697ac08239-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03a7bd96-1a17-4490-8ea8-b0697ac08239" (UID: "03a7bd96-1a17-4490-8ea8-b0697ac08239"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:31.404782 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.404723    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35a494e-db46-4d09-a16b-b2883577e13b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b35a494e-db46-4d09-a16b-b2883577e13b" (UID: "b35a494e-db46-4d09-a16b-b2883577e13b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 11:24:31.406691 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.406665    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a7bd96-1a17-4490-8ea8-b0697ac08239-kube-api-access-q2zkv" (OuterVolumeSpecName: "kube-api-access-q2zkv") pod "03a7bd96-1a17-4490-8ea8-b0697ac08239" (UID: "03a7bd96-1a17-4490-8ea8-b0697ac08239"). InnerVolumeSpecName "kube-api-access-q2zkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:24:31.406781 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.406698    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35a494e-db46-4d09-a16b-b2883577e13b-kube-api-access-nwwhw" (OuterVolumeSpecName: "kube-api-access-nwwhw") pod "b35a494e-db46-4d09-a16b-b2883577e13b" (UID: "b35a494e-db46-4d09-a16b-b2883577e13b"). InnerVolumeSpecName "kube-api-access-nwwhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 11:24:31.505618 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.505594    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b35a494e-db46-4d09-a16b-b2883577e13b-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:31.505618 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.505618    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a7bd96-1a17-4490-8ea8-b0697ac08239-plugins-config-volume\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:31.505766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.505627    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwhw\" (UniqueName: \"kubernetes.io/projected/b35a494e-db46-4d09-a16b-b2883577e13b-kube-api-access-nwwhw\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:31.505766 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.505636    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2zkv\" (UniqueName: \"kubernetes.io/projected/03a7bd96-1a17-4490-8ea8-b0697ac08239-kube-api-access-q2zkv\") on node \"ip-10-0-131-67.ec2.internal\" DevicePath \"\""
Jul 28 11:24:31.588094 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.588058    2578 generic.go:356] "Generic (PLEG): container finished" podID="b35a494e-db46-4d09-a16b-b2883577e13b" containerID="b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c" exitCode=0
Jul 28 11:24:31.588314 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.588111    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"
Jul 28 11:24:31.588314 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.588148    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6" event={"ID":"b35a494e-db46-4d09-a16b-b2883577e13b","Type":"ContainerDied","Data":"b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c"}
Jul 28 11:24:31.588314 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.588200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6" event={"ID":"b35a494e-db46-4d09-a16b-b2883577e13b","Type":"ContainerDied","Data":"ffa0f8463f3091e371797ce3314fa8edee99579b66ce714f4a97faf5e318a453"}
Jul 28 11:24:31.588314 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.588221    2578 scope.go:117] "RemoveContainer" containerID="b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c"
Jul 28 11:24:31.589708 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.589495    2578 generic.go:356] "Generic (PLEG): container finished" podID="03a7bd96-1a17-4490-8ea8-b0697ac08239" containerID="6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444" exitCode=0
Jul 28 11:24:31.589708 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.589542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph" event={"ID":"03a7bd96-1a17-4490-8ea8-b0697ac08239","Type":"ContainerDied","Data":"6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444"}
Jul 28 11:24:31.589708 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.589580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph" event={"ID":"03a7bd96-1a17-4490-8ea8-b0697ac08239","Type":"ContainerDied","Data":"ff534c055837a8b33bbc1eb12a67012397388f3c068de9a0216a59023985c631"}
Jul 28 11:24:31.589708 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.589545    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"
Jul 28 11:24:31.597419 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.597401    2578 scope.go:117] "RemoveContainer" containerID="b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c"
Jul 28 11:24:31.597672 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:24:31.597656    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c\": container with ID starting with b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c not found: ID does not exist" containerID="b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c"
Jul 28 11:24:31.597750 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.597682    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c"} err="failed to get container status \"b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c\": rpc error: code = NotFound desc = could not find container \"b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c\": container with ID starting with b5d1fcae093baf0bfd4a14644ad250777de94ce651a003c4870d42ec83fe577c not found: ID does not exist"
Jul 28 11:24:31.597750 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.597704    2578 scope.go:117] "RemoveContainer" containerID="6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444"
Jul 28 11:24:31.605192 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.605178    2578 scope.go:117] "RemoveContainer" containerID="6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444"
Jul 28 11:24:31.605494 ip-10-0-131-67 kubenswrapper[2578]: E0728 11:24:31.605476    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444\": container with ID starting with 6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444 not found: ID does not exist" containerID="6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444"
Jul 28 11:24:31.605582 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.605505    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444"} err="failed to get container status \"6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444\": rpc error: code = NotFound desc = could not find container \"6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444\": container with ID starting with 6318d18389e08b3a31d6f13ace0052ba7d454d3991619142d409a36025237444 not found: ID does not exist"
Jul 28 11:24:31.634247 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.634215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"]
Jul 28 11:24:31.652889 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.652853    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-683bdc-649dccd896-npkph"]
Jul 28 11:24:31.692793 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.692757    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"]
Jul 28 11:24:31.694339 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:31.694316    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-683bdc-cfc4fb5db-qp8g6"]
Jul 28 11:24:33.493834 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:33.493796    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a7bd96-1a17-4490-8ea8-b0697ac08239" path="/var/lib/kubelet/pods/03a7bd96-1a17-4490-8ea8-b0697ac08239/volumes"
Jul 28 11:24:33.494208 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:33.494158    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b35a494e-db46-4d09-a16b-b2883577e13b" path="/var/lib/kubelet/pods/b35a494e-db46-4d09-a16b-b2883577e13b/volumes"
Jul 28 11:24:34.332984 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:34.332945    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"]
Jul 28 11:24:34.348504 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:34.348468    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-683bdc-vnkqf"]
Jul 28 11:24:35.493854 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:35.493806    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e313909-c157-4fe4-ba40-5966bf2e51f0" path="/var/lib/kubelet/pods/0e313909-c157-4fe4-ba40-5966bf2e51f0/volumes"
Jul 28 11:24:42.819055 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:42.819017    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54bc6dc669-9pw7h_a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c/authorino/0.log"
Jul 28 11:24:49.982684 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:49.982608    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-xxd76_fe736835-a59d-4b26-ab49-1c42127c5344/copy-manifests/0.log"
Jul 28 11:24:49.991992 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:49.991957    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-xxd76_fe736835-a59d-4b26-ab49-1c42127c5344/manager/0.log"
Jul 28 11:24:50.493513 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:50.493487    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753925-jqgk9_ac919f82-e99d-406b-acbd-c95a4beaa4bf/cleanup/0.log"
Jul 28 11:24:50.621027 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:50.620995    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753940-z2bc7_b6058dd3-879d-4378-bfa1-da0cb87d35af/cleanup/0.log"
Jul 28 11:24:50.739439 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:50.739410    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753955-gcj7v_b25fdb20-b2a6-489a-9dd3-7aaf067c9508/cleanup/0.log"
Jul 28 11:24:52.266124 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:52.266087    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54bc6dc669-9pw7h_a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c/authorino/0.log"
Jul 28 11:24:52.395604 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:52.395566    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-sb6jc_7921be03-7225-423e-9007-fd31372340e4/manager/0.log"
Jul 28 11:24:52.513096 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:52.513069    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xkfdg_3fc190f1-a000-4bb3-9d41-0bbabf2d4854/manager/0.log"
Jul 28 11:24:52.636647 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:52.636573    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5jsnm_9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb/kuadrant-console-plugin/0.log"
Jul 28 11:24:53.780631 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:53.780604    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6b86846-5s4d2_41e507ed-b68a-40d0-bf1c-fa97e2005970/kube-auth-proxy/0.log"
Jul 28 11:24:54.914190 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:54.914149    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-vtkvj_d3b6a563-3d5c-4ec4-bd14-fa32a90117e7/main/0.log"
Jul 28 11:24:55.048328 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:24:55.048302    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-k86sz_7753c7ba-d971-4c0d-93e5-14c09b44cdaf/main/0.log"
Jul 28 11:25:01.275368 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275317    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgn8k/must-gather-rnpsm"]
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275641    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e313909-c157-4fe4-ba40-5966bf2e51f0" containerName="revoke-keys"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275652    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e313909-c157-4fe4-ba40-5966bf2e51f0" containerName="revoke-keys"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275672    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275681    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a7bd96-1a17-4490-8ea8-b0697ac08239" containerName="payload-pre-processing"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275687    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a7bd96-1a17-4490-8ea8-b0697ac08239" containerName="payload-pre-processing"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275694    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b35a494e-db46-4d09-a16b-b2883577e13b" containerName="payload-processing"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275700    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b35a494e-db46-4d09-a16b-b2883577e13b" containerName="payload-processing"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275707    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275712    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275761    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b35a494e-db46-4d09-a16b-b2883577e13b" containerName="payload-processing"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275767    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a7bd96-1a17-4490-8ea8-b0697ac08239" containerName="payload-pre-processing"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275774    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275780    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e313909-c157-4fe4-ba40-5966bf2e51f0" containerName="revoke-keys"
Jul 28 11:25:01.275784 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.275786    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b130dafc-5b3d-4a62-bc3d-27087f3d3535" containerName="istio-proxy"
Jul 28 11:25:01.279080 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.279053    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.287315 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.287262    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zgn8k\"/\"default-dockercfg-j5kdx\""
Jul 28 11:25:01.287452 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.287314    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgn8k\"/\"openshift-service-ca.crt\""
Jul 28 11:25:01.288331 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.288313    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zgn8k\"/\"kube-root-ca.crt\""
Jul 28 11:25:01.295604 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.295570    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgn8k/must-gather-rnpsm"]
Jul 28 11:25:01.356167 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.356128    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw2cj\" (UniqueName: \"kubernetes.io/projected/3879ea29-8653-462b-9cb7-590bd88d330f-kube-api-access-pw2cj\") pod \"must-gather-rnpsm\" (UID: \"3879ea29-8653-462b-9cb7-590bd88d330f\") " pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.356396 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.356191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3879ea29-8653-462b-9cb7-590bd88d330f-must-gather-output\") pod \"must-gather-rnpsm\" (UID: \"3879ea29-8653-462b-9cb7-590bd88d330f\") " pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.456912 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.456875    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3879ea29-8653-462b-9cb7-590bd88d330f-must-gather-output\") pod \"must-gather-rnpsm\" (UID: \"3879ea29-8653-462b-9cb7-590bd88d330f\") " pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.457073 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.456955    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw2cj\" (UniqueName: \"kubernetes.io/projected/3879ea29-8653-462b-9cb7-590bd88d330f-kube-api-access-pw2cj\") pod \"must-gather-rnpsm\" (UID: \"3879ea29-8653-462b-9cb7-590bd88d330f\") " pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.457250 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.457229    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3879ea29-8653-462b-9cb7-590bd88d330f-must-gather-output\") pod \"must-gather-rnpsm\" (UID: \"3879ea29-8653-462b-9cb7-590bd88d330f\") " pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.469710 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.469680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw2cj\" (UniqueName: \"kubernetes.io/projected/3879ea29-8653-462b-9cb7-590bd88d330f-kube-api-access-pw2cj\") pod \"must-gather-rnpsm\" (UID: \"3879ea29-8653-462b-9cb7-590bd88d330f\") " pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.589714 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.589610    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgn8k/must-gather-rnpsm"
Jul 28 11:25:01.733494 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.733471    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgn8k/must-gather-rnpsm"]
Jul 28 11:25:01.735940 ip-10-0-131-67 kubenswrapper[2578]: W0728 11:25:01.735904    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3879ea29_8653_462b_9cb7_590bd88d330f.slice/crio-e7c92859967311c260ccd7528465dab05782507ec72b4055c6bf2056b5a7af03 WatchSource:0}: Error finding container e7c92859967311c260ccd7528465dab05782507ec72b4055c6bf2056b5a7af03: Status 404 returned error can't find the container with id e7c92859967311c260ccd7528465dab05782507ec72b4055c6bf2056b5a7af03
Jul 28 11:25:01.737780 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:01.737758    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 11:25:02.740155 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:02.740111    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/must-gather-rnpsm" event={"ID":"3879ea29-8653-462b-9cb7-590bd88d330f","Type":"ContainerStarted","Data":"4db64a58243e889cfb578ee876c987038ac3860c47b2b3606b09d07551d386dd"}
Jul 28 11:25:02.740661 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:02.740175    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/must-gather-rnpsm" event={"ID":"3879ea29-8653-462b-9cb7-590bd88d330f","Type":"ContainerStarted","Data":"e7c92859967311c260ccd7528465dab05782507ec72b4055c6bf2056b5a7af03"}
Jul 28 11:25:03.750130 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:03.750086    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/must-gather-rnpsm" event={"ID":"3879ea29-8653-462b-9cb7-590bd88d330f","Type":"ContainerStarted","Data":"a805e53b600d2e062c300d915621e1786ab737786d022c72c35da7f5a6418000"}
Jul 28 11:25:03.769030 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:03.768959    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgn8k/must-gather-rnpsm" podStartSLOduration=1.8898864789999998 podStartE2EDuration="2.768937161s" totalImagesPullingTime="879.050682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 11:25:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:25:03.76838571 +0000 UTC m=+3694.810477865" watchObservedRunningTime="2026-07-28 11:25:03.768937161 +0000 UTC m=+3694.811029314"
Jul 28 11:25:04.366234 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:04.366199    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d4s8q_081537ec-b941-46e9-b73e-f156ddb7585c/global-pull-secret-syncer/0.log"
Jul 28 11:25:04.591649 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:04.591601    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-h55h6_c7016e81-0f35-49e8-9319-c906df347a4f/konnectivity-agent/0.log"
Jul 28 11:25:04.675836 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:04.675795    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-67.ec2.internal_1b28dbc1bfe8498d00f22a31b5ebf594/haproxy/0.log"
Jul 28 11:25:09.896580 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:09.896204    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-54bc6dc669-9pw7h_a1d0bc1b-0ec8-47ea-ae79-c5ddcc91805c/authorino/0.log"
Jul 28 11:25:09.938255 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:09.938227    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-sb6jc_7921be03-7225-423e-9007-fd31372340e4/manager/0.log"
Jul 28 11:25:09.993774 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:09.993659    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-xkfdg_3fc190f1-a000-4bb3-9d41-0bbabf2d4854/manager/0.log"
Jul 28 11:25:10.030309 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:10.030283    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-5jsnm_9ec1ce85-7bb7-4ae5-abd4-9f1f1a5b60bb/kuadrant-console-plugin/0.log"
Jul 28 11:25:12.520950 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.520913    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-jd9cw_bf3517c1-cacd-4ee7-af11-65d55097e2d5/monitoring-plugin/0.log"
Jul 28 11:25:12.720329 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.720291    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sf288_4951a454-2c35-42c0-a2b3-03e34925ce23/node-exporter/0.log"
Jul 28 11:25:12.743959 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.743922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sf288_4951a454-2c35-42c0-a2b3-03e34925ce23/kube-rbac-proxy/0.log"
Jul 28 11:25:12.768532 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.768435    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sf288_4951a454-2c35-42c0-a2b3-03e34925ce23/init-textfile/0.log"
Jul 28 11:25:12.806822 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.806778    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mpm5v_107932d8-9f04-42e6-8db2-8987be5feb87/kube-rbac-proxy-main/0.log"
Jul 28 11:25:12.837849 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.837806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mpm5v_107932d8-9f04-42e6-8db2-8987be5feb87/kube-rbac-proxy-self/0.log"
Jul 28 11:25:12.869332 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.869296    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-mpm5v_107932d8-9f04-42e6-8db2-8987be5feb87/openshift-state-metrics/0.log"
Jul 28 11:25:12.926904 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.926861    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/prometheus/0.log"
Jul 28 11:25:12.951374 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.951343    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/config-reloader/0.log"
Jul 28 11:25:12.978562 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:12.978525    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/thanos-sidecar/0.log"
Jul 28 11:25:13.008961 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:13.008930    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/kube-rbac-proxy-web/0.log"
Jul 28 11:25:13.033331 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:13.033228    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/kube-rbac-proxy/0.log"
Jul 28 11:25:13.059139 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:13.059068    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/kube-rbac-proxy-thanos/0.log"
Jul 28 11:25:13.085294 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:13.085248    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c42d0cd7-b26b-4ef6-bfcc-d5325387f140/init-config-reloader/0.log"
Jul 28 11:25:13.116971 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:13.116935    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ctxfh_1d917317-ff26-4bfd-9da3-13d4d68761cc/prometheus-operator/0.log"
Jul 28 11:25:13.137053 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:13.137023    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ctxfh_1d917317-ff26-4bfd-9da3-13d4d68761cc/kube-rbac-proxy/0.log"
Jul 28 11:25:14.396554 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.396498    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"]
Jul 28 11:25:14.404510 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.404471    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.411163 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.411126    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"]
Jul 28 11:25:14.492626 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.492582    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-podres\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.492898 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.492881    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-lib-modules\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.493076 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.493057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkhjg\" (UniqueName: \"kubernetes.io/projected/dd985d18-1052-47e5-9f7a-884b71817b3c-kube-api-access-gkhjg\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.493230 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.493215    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-sys\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.493452 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.493438    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-proc\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkhjg\" (UniqueName: \"kubernetes.io/projected/dd985d18-1052-47e5-9f7a-884b71817b3c-kube-api-access-gkhjg\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594119    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-sys\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594187    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-proc\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594237    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-podres\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-lib-modules\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594465    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-lib-modules\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594843    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-sys\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594897    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-proc\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.595014 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.594985    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/dd985d18-1052-47e5-9f7a-884b71817b3c-podres\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.603895 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.603863    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkhjg\" (UniqueName: \"kubernetes.io/projected/dd985d18-1052-47e5-9f7a-884b71817b3c-kube-api-access-gkhjg\") pod \"perf-node-gather-daemonset-dqqnw\" (UID: \"dd985d18-1052-47e5-9f7a-884b71817b3c\") " pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.718627 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.718569    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:14.901099 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:14.901064    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"]
Jul 28 11:25:15.830077 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:15.830031    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-bb7db_3bb5f1bb-5a0c-46da-a128-968330d00d73/download-server/0.log"
Jul 28 11:25:15.837043 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:15.837006    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw" event={"ID":"dd985d18-1052-47e5-9f7a-884b71817b3c","Type":"ContainerStarted","Data":"df4693b91b486f720309186a305c5220d85f0f71e97777f9ab82ab9d74e8a37a"}
Jul 28 11:25:15.837237 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:15.837073    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw" event={"ID":"dd985d18-1052-47e5-9f7a-884b71817b3c","Type":"ContainerStarted","Data":"b69ac8a38cffd55523df1b6ec3988c78321f61089d3a0820d39a8853c6d4fa87"}
Jul 28 11:25:15.837237 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:15.837127    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:15.857207 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:15.857156    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw" podStartSLOduration=1.8571374710000002 podStartE2EDuration="1.857137471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 11:25:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 11:25:15.856789395 +0000 UTC m=+3706.898881553" watchObservedRunningTime="2026-07-28 11:25:15.857137471 +0000 UTC m=+3706.899229627"
Jul 28 11:25:17.547749 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:17.547713    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k4f7h_37c7e810-2dcf-4345-8622-13f6d2e2dbef/dns/0.log"
Jul 28 11:25:17.572462 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:17.572432    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-k4f7h_37c7e810-2dcf-4345-8622-13f6d2e2dbef/kube-rbac-proxy/0.log"
Jul 28 11:25:17.745734 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:17.745702    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q5h4z_24a56284-b1d8-41d0-8f05-a14d507b6e4a/dns-node-resolver/0.log"
Jul 28 11:25:18.400041 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:18.400004    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k44hl_9b4d85bf-16e0-4b73-a1fc-fb1863ef6cfd/node-ca/0.log"
Jul 28 11:25:19.533608 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:19.533579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6b86846-5s4d2_41e507ed-b68a-40d0-bf1c-fa97e2005970/kube-auth-proxy/0.log"
Jul 28 11:25:20.567680 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:20.567651    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gsj9h_d3cc6e7b-8e03-45b4-8853-dd964b1b2f1b/serve-healthcheck-canary/0.log"
Jul 28 11:25:21.256631 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:21.256596    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkhdw_11149244-998e-4872-8941-9fabad25d0a4/kube-rbac-proxy/0.log"
Jul 28 11:25:21.279856 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:21.279829    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkhdw_11149244-998e-4872-8941-9fabad25d0a4/exporter/0.log"
Jul 28 11:25:21.301856 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:21.301806    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-wkhdw_11149244-998e-4872-8941-9fabad25d0a4/extractor/0.log"
Jul 28 11:25:21.857749 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:21.857701    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zgn8k/perf-node-gather-daemonset-dqqnw"
Jul 28 11:25:24.545840 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:24.545750    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-xxd76_fe736835-a59d-4b26-ab49-1c42127c5344/manager/0.log"
Jul 28 11:25:24.571348 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:24.571321    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-559f66f7dc-xxd76_fe736835-a59d-4b26-ab49-1c42127c5344/copy-manifests/0.log"
Jul 28 11:25:25.816666 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:25.816617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7d8dfbcf79-56k9b_9feac2b4-c761-441f-bf84-f8af780e6e44/manager/0.log"
Jul 28 11:25:25.832379 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:25.832350    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6b86846-5s4d2_41e507ed-b68a-40d0-bf1c-fa97e2005970/kube-auth-proxy/0.log"
Jul 28 11:25:27.501098 ip-10-0-131-67 kubenswrapper[2578]: I0728 11:25:27.501073    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d6b86846-5s4d2_41e507ed-b68a-40d0-bf1c-fa97e2005970/kube-auth-proxy/0.log"