Jul 28 19:46:36.983581 ip-10-0-133-160 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 19:46:37.384453 ip-10-0-133-160 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:46:37.384453 ip-10-0-133-160 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 19:46:37.384453 ip-10-0-133-160 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:46:37.384453 ip-10-0-133-160 kubenswrapper[2578]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 19:46:37.384453 ip-10-0-133-160 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 19:46:37.385531 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.385387    2578 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 19:46:37.389879 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389865    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:46:37.389879 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389879    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389884    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389888    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389890    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389893    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389895    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389898    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389901    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389903    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389906    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389908    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389911    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389921    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389924    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389927    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389930    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389932    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389935    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389937    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389940    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:46:37.389941 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389942    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389945    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389948    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389953    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389957    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389960    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389963    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389967    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389970    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389973    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389976    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389979    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389983    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389986    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389989    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389992    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389994    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389997    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.389999    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:46:37.390428 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390002    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390004    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390007    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390009    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390012    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390014    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390017    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390019    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390021    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390024    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390026    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390028    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390031    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390033    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390035    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390039    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390042    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390044    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390047    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390050    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:46:37.390927 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390052    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390054    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390057    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390059    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390062    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390065    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390067    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390069    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390072    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390074    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390077    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390080    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390082    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390084    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390086    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390089    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390092    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390094    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390097    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:46:37.391436 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390101    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390103    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390106    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390108    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390111    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390113    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390116    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390119    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390121    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390124    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390127    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390129    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390132    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390134    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390137    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390139    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390141    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390144    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390146    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.390149    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:46:37.391899 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391009    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391018    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391022    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391024    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391027    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391029    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391032    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391034    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391037    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391040    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391043    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391047    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391050    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391055    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391061    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391064    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391068    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391071    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391073    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:46:37.392393 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391076    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391079    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391081    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391084    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391086    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391089    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391091    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391094    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391096    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391099    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391101    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391104    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391106    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391109    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391111    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391114    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391116    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391118    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391121    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391123    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:46:37.392860 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391126    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391129    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391131    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391134    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391140    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391143    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391145    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391148    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391150    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391153    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391155    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391157    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391160    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391163    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391165    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391168    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391170    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391172    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391175    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391177    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:46:37.393357 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391180    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391183    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391185    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391188    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391190    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391193    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391195    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391198    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391200    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391202    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391205    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391207    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391209    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391212    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391214    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391217    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391221    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391223    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391226    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391229    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:46:37.393857 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391231    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391235    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391238    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391240    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391243    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391246    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391249    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391253    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391256    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391259    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391262    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391285    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391289    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391292    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391295    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391299    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391301    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391304    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391307    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:46:37.394362 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391310    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391696    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391701    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391703    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391706    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391708    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391711    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391714    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391716    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391721    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391724    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391727    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391731    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391735    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391739    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391742    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391745    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391749    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391751    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:46:37.394827 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391754    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391758    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391760    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391763    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391766    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391769    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391772    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391774    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391777    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391779    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391782    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391784    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391787    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391789    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391792    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391795    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391797    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391800    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391802    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:46:37.395338 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391805    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391808    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391810    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391814    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391819    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391822    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391824    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391827    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391830    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391833    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391835    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391838    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391840    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391843    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391845    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391848    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391850    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391853    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391855    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:46:37.395796 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391858    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391860    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391863    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391866    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391868    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391870    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391873    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391876    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391878    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391880    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391884    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391886    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391889    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391891    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391893    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391896    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391900    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391904    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391907    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391910    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:46:37.396260 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391912    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391915    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391917    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391920    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391922    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391925    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391927    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391930    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391932    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391934    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391937    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391939    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391942    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391945    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391947    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391950    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391952    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391955    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391957    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391960    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:46:37.396795 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391962    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391964    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.391967    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392036    2578 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392042    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392049    2578 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392053    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392066    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392070    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392074    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392079    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392083    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392085    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392089    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392092    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392095    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392098    2578 flags.go:64] FLAG: --cgroup-root=""
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392100    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392103    2578 flags.go:64] FLAG: --client-ca-file=""
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392106    2578 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392108    2578 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392112    2578 flags.go:64] FLAG: --cluster-domain=""
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392115    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 19:46:37.397296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392118    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392121    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392124    2578 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392127    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392130    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392133    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392136    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392139    2578 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392142    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392145    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392147    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392150    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392154    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392156    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392159    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392162    2578 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392165    2578 flags.go:64] FLAG: --enable-server="true"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392167    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392172    2578 flags.go:64] FLAG: --event-burst="100"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392176    2578 flags.go:64] FLAG: --event-qps="50"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392179    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392183    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392186    2578 flags.go:64] FLAG: --eviction-hard=""
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392189    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 19:46:37.397897 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392192    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392195    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392198    2578 flags.go:64] FLAG: --eviction-soft=""
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392201    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392203    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392206    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392209    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392212    2578 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392214    2578 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392217    2578 flags.go:64] FLAG: --feature-gates=""
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392221    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392224    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392227    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392230    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392233    2578 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392236    2578 flags.go:64] FLAG: --help="false"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392239    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392242    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392245    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392248    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392251    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392254    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392257    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 19:46:37.398489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392260    2578 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392262    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392285    2578 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392288    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392291    2578 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392294    2578 flags.go:64] FLAG: --kube-reserved=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392297    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392301    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392304    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392307    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392309    2578 flags.go:64] FLAG: --lock-file=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392312    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392315    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392318    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392323    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392326    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392328    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392331    2578 flags.go:64] FLAG: --logging-format="text"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392334    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392337    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392340    2578 flags.go:64] FLAG: --manifest-url=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392343    2578 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392347    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392349    2578 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392354    2578 flags.go:64] FLAG: --max-pods="110"
Jul 28 19:46:37.399059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392357    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392359    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392362    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392365    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392367    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392370    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392373    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392383    2578 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392386    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392389    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392392    2578 flags.go:64] FLAG: --pod-cidr=""
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392395    2578 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392400    2578 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392402    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392405    2578 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392409    2578 flags.go:64] FLAG: --port="10250"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392412    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392415    2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06579709f7596f4ec"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392418    2578 flags.go:64] FLAG: --qos-reserved=""
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392421    2578 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392424    2578 flags.go:64] FLAG: --register-node="true"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392426    2578 flags.go:64] FLAG: --register-with-taints=""
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392430    2578 flags.go:64] FLAG: --registry-burst="10"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392432    2578 flags.go:64] FLAG: --registry-qps="5"
Jul 28 19:46:37.399694 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392435    2578 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392438    2578 flags.go:64] FLAG: --reserved-memory=""
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392441    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392444    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392447    2578 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392449    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392452    2578 flags.go:64] FLAG: --runonce="false"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392456    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392459    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392461    2578 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392464    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392467    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392469    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392472    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392475    2578 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392477    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392480    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392483    2578 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392485    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392490    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392493    2578 flags.go:64] FLAG: --system-cgroups=""
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392496    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392501    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392504    2578 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392507    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 19:46:37.400315 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392511    2578 flags.go:64] FLAG: --tls-min-version=""
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392513    2578 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392516    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392519    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392521    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392524    2578 flags.go:64] FLAG: --v="2"
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392528    2578 flags.go:64] FLAG: --version="false"
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392532    2578 flags.go:64] FLAG: --vmodule=""
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392535    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.392538    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392656    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392661    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392664    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392667    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392671    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392674    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392677    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392679    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392682    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392685    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392687    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:46:37.400934 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392690    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392693    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392696    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392699    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392702    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392708    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392712    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392715    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392718    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392720    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392723    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392726    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392729    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392732    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392735    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392737    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392740    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392742    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392745    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:46:37.401492 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392747    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392750    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392752    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392755    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392757    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392759    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392763    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392766    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392768    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392771    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392773    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392775    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392778    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392780    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392783    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392785    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392788    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392790    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392794    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:46:37.402084 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392796    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392799    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392801    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392804    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392806    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392809    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392812    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392814    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392817    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392820    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392822    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392824    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392827    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392829    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392832    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392834    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392837    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392839    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392841    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392845    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392848    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:46:37.402563 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392850    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392853    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392855    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392858    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392860    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392862    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392865    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392869    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392873    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392875    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392880    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392882    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392885    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392887    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392890    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392893    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392896    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392898    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392902    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:46:37.403073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392904    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392907    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392910    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392912    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392915    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392917    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392920    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392922    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392924    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.392927    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.393588    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.400381    2578 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.400396    2578 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.401113    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.401125    2578 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 19:46:37.403644 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401181    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401186    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401189    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401192    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401195    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401198    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401201    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401204    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401206    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401210    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401214    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401217    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401220    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401222    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401225    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401228    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401231    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401233    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401236    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401238    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:46:37.404013 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401241    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401245    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401249    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401252    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401255    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401258    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401261    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401263    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401280    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401288    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401292    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401295    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401298    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401301    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401304    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401307    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401309    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401312    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401315    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:46:37.404526 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401318    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401320    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401323    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401325    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401328    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401330    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401333    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401336    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401339    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401341    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401344    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401346    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401349    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401351    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401354    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401356    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401359    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401361    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401364    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401366    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:46:37.404979 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401368    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401371    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401374    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401377    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401380    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401382    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401385    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401387    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401390    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401392    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401394    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401398    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401400    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401403    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401406    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401408    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401410    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401413    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401415    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401418    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:46:37.405469 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401420    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401423    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401426    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401428    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401430    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401433    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401435    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401438    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401440    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401442    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401445    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401447    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401450    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401452    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401455    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401457    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401460    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401463    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401465    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:46:37.405964 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401468    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.401473    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401586    2578 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401591    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401595    2578 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401598    2578 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401601    2578 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401603    2578 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401606    2578 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401608    2578 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401611    2578 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401613    2578 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401616    2578 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401619    2578 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 19:46:37.406451 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401622    2578 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401626    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401629    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401632    2578 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401634    2578 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401637    2578 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401639    2578 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401641    2578 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401643    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401646    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401649    2578 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401653    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401655    2578 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401659    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401662    2578 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401665    2578 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401668    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401671    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401673    2578 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 19:46:37.406790 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401676    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401678    2578 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401680    2578 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401683    2578 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401685    2578 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401688    2578 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401690    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401692    2578 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401695    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401698    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401700    2578 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401702    2578 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401705    2578 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401707    2578 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401709    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401712    2578 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401714    2578 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401717    2578 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401719    2578 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401722    2578 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 19:46:37.407256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401724    2578 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401726    2578 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401729    2578 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401731    2578 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401734    2578 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401736    2578 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401739    2578 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401741    2578 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401744    2578 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401746    2578 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401749    2578 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401752    2578 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401754    2578 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401757    2578 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401759    2578 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401762    2578 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401764    2578 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401767    2578 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401769    2578 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401772    2578 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 19:46:37.407754 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401774    2578 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401776    2578 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401779    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401781    2578 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401783    2578 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401786    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401788    2578 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401790    2578 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401793    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401795    2578 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401798    2578 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401800    2578 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401802    2578 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401805    2578 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401807    2578 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401809    2578 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401812    2578 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401815    2578 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401817    2578 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 19:46:37.408496 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401830    2578 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401834    2578 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401837    2578 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401839    2578 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401843    2578 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401845    2578 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401848    2578 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401851    2578 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:37.401853    2578 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.401858    2578 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.401961    2578 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.403925    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.405347    2578 server.go:1019] "Starting client certificate rotation"
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.405461    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 19:46:37.409103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.406209    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 19:46:37.410837 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.410816    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 19:46:37.413483 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.413463    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 19:46:37.421696 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.421680    2578 log.go:25] "Validated CRI v1 runtime API"
Jul 28 19:46:37.425466 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.425451    2578 log.go:25] "Validated CRI v1 image API"
Jul 28 19:46:37.427154 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.427139    2578 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 19:46:37.430927 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.430908    2578 fs.go:135] Filesystem UUIDs: map[1c08336b-47d2-4157-8ccc-e0ad8e37ed53:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 c9ed3758-3e0a-45e9-8eea-0aca279b5fa1:/dev/nvme0n1p3]
Jul 28 19:46:37.430972 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.430927    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 19:46:37.436114 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.436097    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 19:46:37.436362 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.436243    2578 manager.go:216] Machine: {Timestamp:2026-07-28 19:46:37.43495496 +0000 UTC m=+0.335171793 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3071572 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c34296e77791655c91f53664c2781 SystemUUID:ec2c3429-6e77-7916-55c9-1f53664c2781 BootID:385c3d24-1e2d-4145-97f8-2f2a4df3f307 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8d:bd:eb:7e:87 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8d:bd:eb:7e:87 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:60:53:26:44:f7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 19:46:37.436362 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.436360    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 19:46:37.436491 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.436479    2578 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 19:46:37.439465 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.439441    2578 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 19:46:37.439603 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.439467    2578 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-160.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 19:46:37.440188 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.440178    2578 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 19:46:37.440217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.440190    2578 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 19:46:37.440217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.440197    2578 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 19:46:37.440811 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.440801    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 19:46:37.441629 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.441619    2578 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 19:46:37.443442 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443431    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 19:46:37.443565 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443556    2578 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 19:46:37.443596 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443587    2578 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 19:46:37.443626 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443597    2578 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 19:46:37.443626 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443607    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 19:46:37.443626 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443615    2578 kubelet.go:398] "Adding apiserver pod source"
Jul 28 19:46:37.443626 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.443625    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 19:46:37.444106 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.444092    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 19:46:37.444173 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.444115    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 19:46:37.450258 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.450238    2578 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 19:46:37.452338 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.452318    2578 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 19:46:37.452492 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.452469    2578 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 19:46:37.454102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454080    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 19:46:37.454102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454107    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454115    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454122    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454137    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454150    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454159    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454167    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454176    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454193    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454204    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 19:46:37.454213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454209    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 19:46:37.454910 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454901    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 19:46:37.454910 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.454910    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 19:46:37.457453 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.457430    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-160.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 19:46:37.457533 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.457464    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 19:46:37.457533 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.457483    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 19:46:37.458202 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.458191    2578 server.go:1268] "Started kubelet"
Jul 28 19:46:37.458325 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.458292    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 19:46:37.458415 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.458324    2578 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 19:46:37.458415 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.458348    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 19:46:37.459173 ip-10-0-133-160 systemd[1]: Started Kubernetes Kubelet.
Jul 28 19:46:37.460497 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.460482    2578 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 19:46:37.460649 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.460635    2578 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 19:46:37.464126 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.463064    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-160.ec2.internal.18c68bc8f15de61b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-160.ec2.internal,UID:ip-10-0-133-160.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-160.ec2.internal,},FirstTimestamp:2026-07-28 19:46:37.458163227 +0000 UTC m=+0.358380060,LastTimestamp:2026-07-28 19:46:37.458163227 +0000 UTC m=+0.358380060,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-160.ec2.internal,}"
Jul 28 19:46:37.464894 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.464878    2578 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 19:46:37.467729 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.467715    2578 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 19:46:37.467785 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.467734    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 19:46:37.467937 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.467872    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:37.468420 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468402    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 19:46:37.468507 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468407    2578 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 19:46:37.468507 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468440    2578 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 19:46:37.468669 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468654    2578 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 19:46:37.468669 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468670    2578 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 19:46:37.468785 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468766    2578 factory.go:55] Registering systemd factory
Jul 28 19:46:37.468881 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.468813    2578 factory.go:223] Registration of the systemd container factory successfully
Jul 28 19:46:37.469017 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.469000    2578 factory.go:153] Registering CRI-O factory
Jul 28 19:46:37.469082 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.469024    2578 factory.go:223] Registration of the crio container factory successfully
Jul 28 19:46:37.469113 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.469090    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 19:46:37.469145 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.469116    2578 factory.go:103] Registering Raw factory
Jul 28 19:46:37.469145 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.469133    2578 manager.go:1196] Started watching for new ooms in manager
Jul 28 19:46:37.469677 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.469662    2578 manager.go:349] Starting recovery of all containers
Jul 28 19:46:37.478408 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.478380    2578 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 19:46:37.478533 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.478511    2578 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-160.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 19:46:37.479239 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.479214    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-457p8"
Jul 28 19:46:37.482303 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.482152    2578 manager.go:354] Recovery completed
Jul 28 19:46:37.486103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.486076    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-457p8"
Jul 28 19:46:37.488058 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.488043    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.490015 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.489992    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.490179 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.490032    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.490179 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.490047    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.490723 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.490710    2578 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 19:46:37.490723 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.490721    2578 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 19:46:37.490723 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.490725    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.490930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.490735    2578 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 19:46:37.491536 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.491518    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.491622 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.491555    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.491622 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.491566    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.492396 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.492384    2578 policy_none.go:49] "None policy: Start"
Jul 28 19:46:37.492444 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.492400    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 19:46:37.492444 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.492410    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 19:46:37.493414 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.493404    2578 policy_none.go:47] "Start"
Jul 28 19:46:37.541855 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.541836    2578 manager.go:337] "Starting Device Plugin manager"
Jul 28 19:46:37.541942 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.541872    2578 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 19:46:37.541942 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.541882    2578 server.go:87] "Starting device plugin registration server"
Jul 28 19:46:37.542199 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.542182    2578 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 19:46:37.542285 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.542200    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 19:46:37.542357 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.542341    2578 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 19:46:37.542907 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.542430    2578 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 19:46:37.542907 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.542444    2578 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 19:46:37.543039 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.542990    2578 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 19:46:37.543097 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.543040    2578 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:37.642599 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.642534    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.643093 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.643058    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 19:46:37.643697 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.643677    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.643820 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.643707    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.643820 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.643718    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.643820 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.643744    2578 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.644489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.644471    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 19:46:37.644578 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.644518    2578 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 19:46:37.644578 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.644546    2578 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 19:46:37.644684 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.644587    2578 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 19:46:37.646775 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.646755    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:46:37.650671 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.650652    2578 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.650746 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.650673    2578 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-160.ec2.internal\": node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:37.693261 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.693235    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:37.745072 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.745031    2578 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"]
Jul 28 19:46:37.745172 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.745138    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.746705 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.746690    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.746785 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.746719    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.746785 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.746734    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.748098 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748085    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.748213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.748296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748241    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.748809 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748795    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.748877 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748804    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.748877 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748822    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.748877 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748826    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.748877 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748838    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.748877 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.748848    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.750177 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.750161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.750258 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.750183    2578 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 19:46:37.750837 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.750821    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 19:46:37.750921 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.750848    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 19:46:37.750921 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.750862    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeHasSufficientPID"
Jul 28 19:46:37.769945 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.769923    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cabbd4b8e7f726e6a0470a46a65225fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"cabbd4b8e7f726e6a0470a46a65225fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.770035 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.769948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabbd4b8e7f726e6a0470a46a65225fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"cabbd4b8e7f726e6a0470a46a65225fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.770035 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.769968    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e70b54208e499ae364c57edb3340d4b8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-160.ec2.internal\" (UID: \"e70b54208e499ae364c57edb3340d4b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.779101 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.779081    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-160.ec2.internal\" not found" node="ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.783659 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.783642    2578 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-160.ec2.internal\" not found" node="ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.793936 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.793916    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:37.870367 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.870334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cabbd4b8e7f726e6a0470a46a65225fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"cabbd4b8e7f726e6a0470a46a65225fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.870367 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.870371    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabbd4b8e7f726e6a0470a46a65225fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"cabbd4b8e7f726e6a0470a46a65225fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.870571 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.870389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e70b54208e499ae364c57edb3340d4b8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-160.ec2.internal\" (UID: \"e70b54208e499ae364c57edb3340d4b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.870571 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.870424    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e70b54208e499ae364c57edb3340d4b8-config\") pod \"kube-apiserver-proxy-ip-10-0-133-160.ec2.internal\" (UID: \"e70b54208e499ae364c57edb3340d4b8\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.870571 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.870410    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/cabbd4b8e7f726e6a0470a46a65225fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"cabbd4b8e7f726e6a0470a46a65225fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.870571 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:37.870429    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cabbd4b8e7f726e6a0470a46a65225fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal\" (UID: \"cabbd4b8e7f726e6a0470a46a65225fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:37.894514 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.894443    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:37.995233 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:37.995200    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.081762 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.081737    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:38.086737 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.086719    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:38.095711 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.095690    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.196318 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.196218    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.296799 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.296765    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.397406 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.397374    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.404552 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.404528    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 19:46:38.404686 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.404669    2578 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 19:46:38.490984 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.490900    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 19:41:38 +0000 UTC" deadline="2028-03-07 00:33:22.03169644 +0000 UTC"
Jul 28 19:46:38.490984 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.490929    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14092h46m43.540770387s"
Jul 28 19:46:38.498061 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.498038    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.598718 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.598686    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.628748 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.628718    2578 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:46:38.665741 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:38.665702    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70b54208e499ae364c57edb3340d4b8.slice/crio-b1178367f2ed3a7c705196ffeda2f72e7dade4a62994ce116e4715072cc73e6f WatchSource:0}: Error finding container b1178367f2ed3a7c705196ffeda2f72e7dade4a62994ce116e4715072cc73e6f: Status 404 returned error can't find the container with id b1178367f2ed3a7c705196ffeda2f72e7dade4a62994ce116e4715072cc73e6f
Jul 28 19:46:38.666000 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:38.665972    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabbd4b8e7f726e6a0470a46a65225fe.slice/crio-0b01dbf45f786b101ed3824b0fe832e852ccc767d355c06ba6913407911ad954 WatchSource:0}: Error finding container 0b01dbf45f786b101ed3824b0fe832e852ccc767d355c06ba6913407911ad954: Status 404 returned error can't find the container with id 0b01dbf45f786b101ed3824b0fe832e852ccc767d355c06ba6913407911ad954
Jul 28 19:46:38.670148 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.670134    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:46:38.699377 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.699348    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.799942 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:38.799866    2578 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-160.ec2.internal\" not found"
Jul 28 19:46:38.816172 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.816140    2578 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:46:38.818134 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.818117    2578 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:46:38.868055 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.868030    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:38.883296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.883277    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 19:46:38.884465 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.884452    2578 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal"
Jul 28 19:46:38.890186 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:38.890174    2578 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 19:46:39.444772 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.444738    2578 apiserver.go:52] "Watching apiserver"
Jul 28 19:46:39.453867 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.453841    2578 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 19:46:39.456257 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.456226    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-hh7c9","kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v","openshift-cluster-node-tuning-operator/tuned-phkfk","openshift-image-registry/node-ca-974jf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal","openshift-multus/multus-additional-cni-plugins-knqbt","openshift-dns/node-resolver-xx69x","openshift-multus/multus-qljrk","openshift-multus/network-metrics-daemon-m9tq9","openshift-network-diagnostics/network-check-target-n995h","openshift-network-operator/iptables-alerter-zxx52","openshift-ovn-kubernetes/ovnkube-node-cbv5w","kube-system/global-pull-secret-syncer-lt7fb"]
Jul 28 19:46:39.457983 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.457956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.459071 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.459048    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:39.459166 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.459119    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:39.460305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.460284    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 19:46:39.460458 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.460435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.461023 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.460986    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.461023 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.461015    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 19:46:39.461177 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.461041    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-gtzzl\""
Jul 28 19:46:39.461177 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.461089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.461177 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.461089    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 19:46:39.461963 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.461942    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.462855 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.462832    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-bbjrs\""
Jul 28 19:46:39.462949 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.462880    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.463680 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.463297    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 19:46:39.463680 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.463350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.463680 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.463373    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.464021 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.464005    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.464173 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.464156    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.464328 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.464308    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kj677\""
Jul 28 19:46:39.464702 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.464685    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.465470 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.465445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.465559 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.465531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jh7gg\""
Jul 28 19:46:39.465810 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.465793    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.466773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.466145    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.466902 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.466884    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.467887 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.467861    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 19:46:39.467887 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.467877    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ptpmw\""
Jul 28 19:46:39.467993 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.467914    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.468180 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.468157    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.468827 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.468435    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fjrsb\""
Jul 28 19:46:39.468827 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.468640    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 19:46:39.468827 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.468740    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 19:46:39.469944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.469924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:39.470040 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.469999    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:39.470440 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.470421    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 19:46:39.470535 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.470483    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 19:46:39.470808 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.470787    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 19:46:39.471089 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.471067    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.471187 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.471167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2qqq\""
Jul 28 19:46:39.471307 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.471258    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 19:46:39.471724 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.471709    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.472217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.472200    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.473260 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.473131    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.473953 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.473935    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 19:46:39.474038 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.473991    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 19:46:39.474038 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.474000    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dmqfh\""
Jul 28 19:46:39.474485 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.474457    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.474522 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.474506    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:39.475339 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.475317    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 19:46:39.475444 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.475428    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 19:46:39.475490 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.475326    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 19:46:39.475685 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.475670    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8xhmz\""
Jul 28 19:46:39.478644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478621    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fls64\" (UniqueName: \"kubernetes.io/projected/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-kube-api-access-fls64\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.478738 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478653    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-cni-binary-copy\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.478738 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478676    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.478738 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478702    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-registration-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.478738 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478725    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-hostroot\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysctl-conf\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478811    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-kubelet\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478852    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-sys-fs\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-modprobe-d\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478926    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntfq\" (UniqueName: \"kubernetes.io/projected/a37fb47a-b3ea-40cf-90e2-28e48e50a234-kube-api-access-zntfq\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.478952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478949    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-cni-netd\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.478989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479011    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-cni-bin\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479032    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-os-release\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479056    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysconfig\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-systemd-units\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479116    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovnkube-config\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479137    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovnkube-script-lib\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479160    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479191    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-socket-dir-parent\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-host\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479236    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-conf-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479256    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-sys\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479326    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovn-node-metrics-cert\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479376    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-cni-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479418    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-os-release\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-k8s-cni-cncf-io\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479473    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-etc-kubernetes\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-node-log\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479539    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8qk\" (UniqueName: \"kubernetes.io/projected/59603633-ac5f-4664-b137-310733f393f9-kube-api-access-6w8qk\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479593    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysctl-d\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479625    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-cni-bin\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479650    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-kubelet\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479682    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37fb47a-b3ea-40cf-90e2-28e48e50a234-iptables-alerter-script\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-run-netns\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.479817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-env-overrides\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-cnibin\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479825    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-netns\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479859    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-cni-multus\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-ovn\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-socket-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479941    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-multus-certs\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479963    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77e8b7e-663f-417e-8e2c-73683e752699-tmp-dir\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.479985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-etc-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480006    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-var-lib-kubelet\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-slash\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480313    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4snrl\" (UniqueName: \"kubernetes.io/projected/89149230-3f6c-41e2-a670-626669e37bd9-kube-api-access-4snrl\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.480581 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480368    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-device-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.481227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb0afac4-b83a-4b2a-9942-35a729592390-multus-daemon-config\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.481227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-system-cni-dir\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.481227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480768    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-system-cni-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.481227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.480800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7cnt\" (UniqueName: \"kubernetes.io/projected/eb0afac4-b83a-4b2a-9942-35a729592390-kube-api-access-j7cnt\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.481227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481107    2578 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 19:46:39.481780 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481759    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ltr9\" (UniqueName: \"kubernetes.io/projected/03c8c9fd-d068-468d-a118-e2a49e136cee-kube-api-access-7ltr9\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:39.481870 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481810    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-lib-modules\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.481870 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481850    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-tmp\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.481978 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pg6m\" (UniqueName: \"kubernetes.io/projected/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-kube-api-access-2pg6m\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.481978 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481914    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37fb47a-b3ea-40cf-90e2-28e48e50a234-host-slash\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.481978 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481942    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-systemd\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.481978 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-log-socket\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.481992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-cnibin\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb0afac4-b83a-4b2a-9942-35a729592390-cni-binary-copy\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482096    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482123    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-kubernetes\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-systemd\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-run\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482196    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-tuned\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.482253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482222    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e77e8b7e-663f-417e-8e2c-73683e752699-hosts-file\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.482789 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482250    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6t2n\" (UniqueName: \"kubernetes.io/projected/e77e8b7e-663f-417e-8e2c-73683e752699-kube-api-access-k6t2n\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.482789 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.482298    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-var-lib-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.500982 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.500964    2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p77tl"
Jul 28 19:46:39.508176 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.508154    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p77tl"
Jul 28 19:46:39.569494 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.569454    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 19:46:39.582494 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582467    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.582494 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovn-node-metrics-cert\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582511    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582528    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-cni-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582542    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-os-release\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-k8s-cni-cncf-io\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582576    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-etc-kubernetes\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582621    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-os-release\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582661    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-k8s-cni-cncf-io\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582627    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-etc-kubernetes\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582684    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cad5271-713f-4162-b9ad-00cdc9dd0221-kubelet-config\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.582712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582712    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-node-log\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-cni-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582734    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8qk\" (UniqueName: \"kubernetes.io/projected/59603633-ac5f-4664-b137-310733f393f9-kube-api-access-6w8qk\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582778    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-node-log\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysctl-d\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582872    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-cni-bin\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582896    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysctl-d\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582904    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-cni-bin\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-etc-selinux\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-kubelet\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582928    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-kubelet\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582930    2578 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582944    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cad5271-713f-4162-b9ad-00cdc9dd0221-dbus\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582973    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37fb47a-b3ea-40cf-90e2-28e48e50a234-iptables-alerter-script\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.582995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-run-netns\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583017    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-env-overrides\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583037    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-cnibin\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.583217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583075    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-netns\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-cni-multus\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-ovn\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583134    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-cnibin\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583091    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-run-netns\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583185    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-ovn\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583191    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-netns\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583137    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-socket-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583248    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-multus-certs\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583292    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77e8b7e-663f-417e-8e2c-73683e752699-tmp-dir\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583248    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-socket-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583301    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-cni-multus\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-run-multus-certs\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583343    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/986541f9-ad8a-4755-b6bc-37e010abdbdb-serviceca\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583374    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-etc-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583398    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-var-lib-kubelet\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583421    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-slash\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.584081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-etc-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583471    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583496    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4snrl\" (UniqueName: \"kubernetes.io/projected/89149230-3f6c-41e2-a670-626669e37bd9-kube-api-access-4snrl\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583508    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-var-lib-kubelet\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583520    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-device-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb0afac4-b83a-4b2a-9942-35a729592390-multus-daemon-config\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-run-ovn-kubernetes\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583562    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a37fb47a-b3ea-40cf-90e2-28e48e50a234-iptables-alerter-script\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583572    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e77e8b7e-663f-417e-8e2c-73683e752699-tmp-dir\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583583    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-slash\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zdhk\" (UniqueName: \"kubernetes.io/projected/986541f9-ad8a-4755-b6bc-37e010abdbdb-kube-api-access-9zdhk\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-env-overrides\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583646    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-device-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583710    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-system-cni-dir\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-system-cni-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7cnt\" (UniqueName: \"kubernetes.io/projected/eb0afac4-b83a-4b2a-9942-35a729592390-kube-api-access-j7cnt\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583771    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-system-cni-dir\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.585095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ltr9\" (UniqueName: \"kubernetes.io/projected/03c8c9fd-d068-468d-a118-e2a49e136cee-kube-api-access-7ltr9\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583815    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-system-cni-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583922    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-lib-modules\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.583964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-tmp\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584026    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pg6m\" (UniqueName: \"kubernetes.io/projected/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-kube-api-access-2pg6m\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eb0afac4-b83a-4b2a-9942-35a729592390-multus-daemon-config\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37fb47a-b3ea-40cf-90e2-28e48e50a234-host-slash\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584086    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-lib-modules\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584114    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a37fb47a-b3ea-40cf-90e2-28e48e50a234-host-slash\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-systemd\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584219    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-log-socket\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584247    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584249    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-run-systemd\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584296    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-log-socket\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584291    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-cnibin\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584334    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb0afac4-b83a-4b2a-9942-35a729592390-cni-binary-copy\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584358    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:39.586026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584372    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-cnibin\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584384    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-kubernetes\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584408    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-systemd\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584432    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-run\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.584451    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584473    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-kubernetes\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-tuned\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.584523    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:46:40.084501881 +0000 UTC m=+2.984718727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584545    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e77e8b7e-663f-417e-8e2c-73683e752699-hosts-file\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584573    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6t2n\" (UniqueName: \"kubernetes.io/projected/e77e8b7e-663f-417e-8e2c-73683e752699-kube-api-access-k6t2n\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-var-lib-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fls64\" (UniqueName: \"kubernetes.io/projected/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-kube-api-access-fls64\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-cni-binary-copy\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e77e8b7e-663f-417e-8e2c-73683e752699-hosts-file\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584813    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-registration-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-hostroot\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.586944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584837    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eb0afac4-b83a-4b2a-9942-35a729592390-cni-binary-copy\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-systemd\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584863    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysctl-conf\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584883    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-var-lib-openvswitch\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584892    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-kubelet\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584911    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-run\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584921    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584968    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-sys-fs\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.584992    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-modprobe-d\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585020    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41ca7edc-a640-4fa8-97db-f7b908e53ed1-konnectivity-ca\") pod \"konnectivity-agent-hh7c9\" (UID: \"41ca7edc-a640-4fa8-97db-f7b908e53ed1\") " pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntfq\" (UniqueName: \"kubernetes.io/projected/a37fb47a-b3ea-40cf-90e2-28e48e50a234-kube-api-access-zntfq\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585073    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysctl-conf\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-cni-netd\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-cni-netd\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585153    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-cni-bin\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585182    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41ca7edc-a640-4fa8-97db-f7b908e53ed1-agent-certs\") pod \"konnectivity-agent-hh7c9\" (UID: \"41ca7edc-a640-4fa8-97db-f7b908e53ed1\") " pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.587587 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585192    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-cni-binary-copy\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-os-release\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585231    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysconfig\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585253    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/986541f9-ad8a-4755-b6bc-37e010abdbdb-host\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585298    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-systemd-units\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585324    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovnkube-config\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585351    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585357    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovnkube-script-lib\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585402    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585429    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-socket-dir-parent\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585442    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-hostroot\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585454    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-host\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585480    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-conf-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-sys\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585570    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585602    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-sys\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585643    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89149230-3f6c-41e2-a670-626669e37bd9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-socket-dir-parent\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585700    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-host-kubelet\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-host\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-registration-dir\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585792    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-multus-conf-dir\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585804    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/59603633-ac5f-4664-b137-310733f393f9-sys-fs\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585870    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eb0afac4-b83a-4b2a-9942-35a729592390-host-var-lib-cni-bin\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovnkube-script-lib\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585905    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-modprobe-d\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585908    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-systemd-units\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/89149230-3f6c-41e2-a670-626669e37bd9-os-release\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.585951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-sysconfig\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.586776    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovn-node-metrics-cert\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.586969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-ovnkube-config\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.587163    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-etc-tuned\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.588623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.587672    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-tmp\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.591698 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.591667    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8qk\" (UniqueName: \"kubernetes.io/projected/59603633-ac5f-4664-b137-310733f393f9-kube-api-access-6w8qk\") pod \"aws-ebs-csi-driver-node-8nq4v\" (UID: \"59603633-ac5f-4664-b137-310733f393f9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.591808 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.591765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7cnt\" (UniqueName: \"kubernetes.io/projected/eb0afac4-b83a-4b2a-9942-35a729592390-kube-api-access-j7cnt\") pod \"multus-qljrk\" (UID: \"eb0afac4-b83a-4b2a-9942-35a729592390\") " pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.592249 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.592230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4snrl\" (UniqueName: \"kubernetes.io/projected/89149230-3f6c-41e2-a670-626669e37bd9-kube-api-access-4snrl\") pod \"multus-additional-cni-plugins-knqbt\" (UID: \"89149230-3f6c-41e2-a670-626669e37bd9\") " pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.592412 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.592388    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ltr9\" (UniqueName: \"kubernetes.io/projected/03c8c9fd-d068-468d-a118-e2a49e136cee-kube-api-access-7ltr9\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:39.594817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.594783    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pg6m\" (UniqueName: \"kubernetes.io/projected/9eb2af8b-a861-4a81-875e-e4ffb9f7bb94-kube-api-access-2pg6m\") pod \"tuned-phkfk\" (UID: \"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94\") " pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.596555 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.596536    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:46:39.596555 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.596556    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:46:39.597212 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.596565    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:39.597212 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.596620    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:40.096604694 +0000 UTC m=+2.996821522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:39.598380 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.598298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6t2n\" (UniqueName: \"kubernetes.io/projected/e77e8b7e-663f-417e-8e2c-73683e752699-kube-api-access-k6t2n\") pod \"node-resolver-xx69x\" (UID: \"e77e8b7e-663f-417e-8e2c-73683e752699\") " pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.598738 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.598721    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntfq\" (UniqueName: \"kubernetes.io/projected/a37fb47a-b3ea-40cf-90e2-28e48e50a234-kube-api-access-zntfq\") pod \"iptables-alerter-zxx52\" (UID: \"a37fb47a-b3ea-40cf-90e2-28e48e50a234\") " pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.599104 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.599085    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fls64\" (UniqueName: \"kubernetes.io/projected/e4f7328e-2111-43f3-a7bc-f47113f1c2a5-kube-api-access-fls64\") pod \"ovnkube-node-cbv5w\" (UID: \"e4f7328e-2111-43f3-a7bc-f47113f1c2a5\") " pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.652857 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.652793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal" event={"ID":"e70b54208e499ae364c57edb3340d4b8","Type":"ContainerStarted","Data":"b1178367f2ed3a7c705196ffeda2f72e7dade4a62994ce116e4715072cc73e6f"}
Jul 28 19:46:39.654371 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.654338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" event={"ID":"cabbd4b8e7f726e6a0470a46a65225fe","Type":"ContainerStarted","Data":"0b01dbf45f786b101ed3824b0fe832e852ccc767d355c06ba6913407911ad954"}
Jul 28 19:46:39.686544 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.686483    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41ca7edc-a640-4fa8-97db-f7b908e53ed1-konnectivity-ca\") pod \"konnectivity-agent-hh7c9\" (UID: \"41ca7edc-a640-4fa8-97db-f7b908e53ed1\") " pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.686544 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.686541    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41ca7edc-a640-4fa8-97db-f7b908e53ed1-agent-certs\") pod \"konnectivity-agent-hh7c9\" (UID: \"41ca7edc-a640-4fa8-97db-f7b908e53ed1\") " pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.687049 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687023    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/986541f9-ad8a-4755-b6bc-37e010abdbdb-host\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.687136 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/41ca7edc-a640-4fa8-97db-f7b908e53ed1-konnectivity-ca\") pod \"konnectivity-agent-hh7c9\" (UID: \"41ca7edc-a640-4fa8-97db-f7b908e53ed1\") " pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.687136 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687085    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cad5271-713f-4162-b9ad-00cdc9dd0221-kubelet-config\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.687136 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687131    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cad5271-713f-4162-b9ad-00cdc9dd0221-dbus\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.687314 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687158    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4cad5271-713f-4162-b9ad-00cdc9dd0221-kubelet-config\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.687314 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687160    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/986541f9-ad8a-4755-b6bc-37e010abdbdb-serviceca\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.687314 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.687314 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687234    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zdhk\" (UniqueName: \"kubernetes.io/projected/986541f9-ad8a-4755-b6bc-37e010abdbdb-kube-api-access-9zdhk\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.687595 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687563    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/986541f9-ad8a-4755-b6bc-37e010abdbdb-serviceca\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.687715 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687626    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/986541f9-ad8a-4755-b6bc-37e010abdbdb-host\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.687715 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.687694    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:39.687715 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.687698    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4cad5271-713f-4162-b9ad-00cdc9dd0221-dbus\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:39.687891 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:39.687742    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:40.187728074 +0000 UTC m=+3.087944895 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:39.690002 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.689969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/41ca7edc-a640-4fa8-97db-f7b908e53ed1-agent-certs\") pod \"konnectivity-agent-hh7c9\" (UID: \"41ca7edc-a640-4fa8-97db-f7b908e53ed1\") " pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.696925 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.696844    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zdhk\" (UniqueName: \"kubernetes.io/projected/986541f9-ad8a-4755-b6bc-37e010abdbdb-kube-api-access-9zdhk\") pod \"node-ca-974jf\" (UID: \"986541f9-ad8a-4755-b6bc-37e010abdbdb\") " pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.770217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.770183    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-knqbt"
Jul 28 19:46:39.778525 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.778492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v"
Jul 28 19:46:39.786541 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.786515    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-phkfk"
Jul 28 19:46:39.792657 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.792629    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xx69x"
Jul 28 19:46:39.800044 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.800020    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxx52"
Jul 28 19:46:39.807046 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.807022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-qljrk"
Jul 28 19:46:39.813453 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.813419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:46:39.822401 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.822370    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:46:39.826798 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.826758    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-974jf"
Jul 28 19:46:39.890489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:39.890445    2578 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 19:46:40.091415 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:40.091325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:40.091549 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.091518    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:40.091615 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.091603    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:46:41.091581063 +0000 UTC m=+3.991797904 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:40.192163 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:40.192121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:40.192365 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:40.192208    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:40.192365 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.192340    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:40.192499 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.192381    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:46:40.192499 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.192399    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:46:40.192499 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.192409    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:40.192499 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.192428    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:41.192405132 +0000 UTC m=+4.092621965 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:40.192499 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.192458    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:41.192440274 +0000 UTC m=+4.092657097 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:40.510928 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:40.510877    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:41:40 +0000 UTC" deadline="2028-03-16 04:51:15.557055684 +0000 UTC"
Jul 28 19:46:40.510928 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:40.510923    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14313h4m35.046139621s"
Jul 28 19:46:40.645662 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:40.645621    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:40.645823 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:40.645755    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:40.755272 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.755022    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f7328e_2111_43f3_a7bc_f47113f1c2a5.slice/crio-4d961c02199535459715368f56a2028f957a59dbf54231427cb667a329f0770a WatchSource:0}: Error finding container 4d961c02199535459715368f56a2028f957a59dbf54231427cb667a329f0770a: Status 404 returned error can't find the container with id 4d961c02199535459715368f56a2028f957a59dbf54231427cb667a329f0770a
Jul 28 19:46:40.757421 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.757393    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59603633_ac5f_4664_b137_310733f393f9.slice/crio-d2e16a1ce0859f3ee0c6713b611f6b92240e5ea5a012da8f5735c3eb10432510 WatchSource:0}: Error finding container d2e16a1ce0859f3ee0c6713b611f6b92240e5ea5a012da8f5735c3eb10432510: Status 404 returned error can't find the container with id d2e16a1ce0859f3ee0c6713b611f6b92240e5ea5a012da8f5735c3eb10432510
Jul 28 19:46:40.761196 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.761162    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89149230_3f6c_41e2_a670_626669e37bd9.slice/crio-e9d065124ce9768519125d886d3ef7a24195922c10e478f0ab25a55ef93df4d1 WatchSource:0}: Error finding container e9d065124ce9768519125d886d3ef7a24195922c10e478f0ab25a55ef93df4d1: Status 404 returned error can't find the container with id e9d065124ce9768519125d886d3ef7a24195922c10e478f0ab25a55ef93df4d1
Jul 28 19:46:40.761904 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.761882    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda37fb47a_b3ea_40cf_90e2_28e48e50a234.slice/crio-f646b78327f7bbded86b0eb73519255cd685d18bcb53b4cccc02d296da97aff3 WatchSource:0}: Error finding container f646b78327f7bbded86b0eb73519255cd685d18bcb53b4cccc02d296da97aff3: Status 404 returned error can't find the container with id f646b78327f7bbded86b0eb73519255cd685d18bcb53b4cccc02d296da97aff3
Jul 28 19:46:40.764113 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.763819    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0afac4_b83a_4b2a_9942_35a729592390.slice/crio-31e153b79d3607d756a5cdde1de91c3afa0c070fe08d65da0329a000429d32f1 WatchSource:0}: Error finding container 31e153b79d3607d756a5cdde1de91c3afa0c070fe08d65da0329a000429d32f1: Status 404 returned error can't find the container with id 31e153b79d3607d756a5cdde1de91c3afa0c070fe08d65da0329a000429d32f1
Jul 28 19:46:40.765816 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.765760    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod986541f9_ad8a_4755_b6bc_37e010abdbdb.slice/crio-ff784c28397122c690675cc6938e71764972b3548177ccad0d7fb43b0ad2ac90 WatchSource:0}: Error finding container ff784c28397122c690675cc6938e71764972b3548177ccad0d7fb43b0ad2ac90: Status 404 returned error can't find the container with id ff784c28397122c690675cc6938e71764972b3548177ccad0d7fb43b0ad2ac90
Jul 28 19:46:40.770216 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:46:40.767954    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77e8b7e_663f_417e_8e2c_73683e752699.slice/crio-7e8bd1d70a118841bf38be9c26f12cd0e72a1b29ca19cb323047f338885c28e0 WatchSource:0}: Error finding container 7e8bd1d70a118841bf38be9c26f12cd0e72a1b29ca19cb323047f338885c28e0: Status 404 returned error can't find the container with id 7e8bd1d70a118841bf38be9c26f12cd0e72a1b29ca19cb323047f338885c28e0
Jul 28 19:46:41.099441 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.099314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:41.099589 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.099462    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:41.099589 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.099532    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:46:43.099512948 +0000 UTC m=+5.999729771 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:41.200402 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.200369    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:41.200578 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.200424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:41.200578 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.200546    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:41.200578 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.200566    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:46:41.200687 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.200581    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:46:41.200687 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.200591    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:41.200687 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.200620    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:43.200600506 +0000 UTC m=+6.100817335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:41.200687 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.200638    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:43.200629684 +0000 UTC m=+6.100846505 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:41.511750 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.511647    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 19:41:40 +0000 UTC" deadline="2028-02-24 16:56:34.867313852 +0000 UTC"
Jul 28 19:46:41.511750 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.511696    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13821h9m53.355623729s"
Jul 28 19:46:41.647592 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.647552    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:41.647793 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.647728    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:41.648173 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.648150    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:41.648280 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:41.648259    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:41.672939 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.672895    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxx52" event={"ID":"a37fb47a-b3ea-40cf-90e2-28e48e50a234","Type":"ContainerStarted","Data":"f646b78327f7bbded86b0eb73519255cd685d18bcb53b4cccc02d296da97aff3"}
Jul 28 19:46:41.678368 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.678333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v" event={"ID":"59603633-ac5f-4664-b137-310733f393f9","Type":"ContainerStarted","Data":"d2e16a1ce0859f3ee0c6713b611f6b92240e5ea5a012da8f5735c3eb10432510"}
Jul 28 19:46:41.692173 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.692136    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal" event={"ID":"e70b54208e499ae364c57edb3340d4b8","Type":"ContainerStarted","Data":"8f1b8f0b07c6b046b4581279659690013a1c15468e9e5d44cb6c405c0b228e90"}
Jul 28 19:46:41.702338 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.702286    2578 generic.go:356] "Generic (PLEG): container finished" podID="cabbd4b8e7f726e6a0470a46a65225fe" containerID="93f06d85121f14e7e5b5b2cb3f049c0ad96878b684f4d2b2e22d198d2e1ef038" exitCode=0
Jul 28 19:46:41.702475 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.702368    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" event={"ID":"cabbd4b8e7f726e6a0470a46a65225fe","Type":"ContainerDied","Data":"93f06d85121f14e7e5b5b2cb3f049c0ad96878b684f4d2b2e22d198d2e1ef038"}
Jul 28 19:46:41.715178 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.715093    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-160.ec2.internal" podStartSLOduration=2.715054699 podStartE2EDuration="2.715054699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:41.712762623 +0000 UTC m=+4.612979469" watchObservedRunningTime="2026-07-28 19:46:41.715054699 +0000 UTC m=+4.615271546"
Jul 28 19:46:41.716310 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.716275    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phkfk" event={"ID":"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94","Type":"ContainerStarted","Data":"347a7877e85f0518eacb9987b0b6f14c387b26baa5c1c4e85d57664436e7d7af"}
Jul 28 19:46:41.726447 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.726419    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hh7c9" event={"ID":"41ca7edc-a640-4fa8-97db-f7b908e53ed1","Type":"ContainerStarted","Data":"d983455e419c4782687a760903d15da82ada58587804e4f6cab31b468d787625"}
Jul 28 19:46:41.733514 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.733481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xx69x" event={"ID":"e77e8b7e-663f-417e-8e2c-73683e752699","Type":"ContainerStarted","Data":"7e8bd1d70a118841bf38be9c26f12cd0e72a1b29ca19cb323047f338885c28e0"}
Jul 28 19:46:41.735866 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.735832    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerStarted","Data":"e9d065124ce9768519125d886d3ef7a24195922c10e478f0ab25a55ef93df4d1"}
Jul 28 19:46:41.739948 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.739921    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"4d961c02199535459715368f56a2028f957a59dbf54231427cb667a329f0770a"}
Jul 28 19:46:41.765960 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.765883    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-974jf" event={"ID":"986541f9-ad8a-4755-b6bc-37e010abdbdb","Type":"ContainerStarted","Data":"ff784c28397122c690675cc6938e71764972b3548177ccad0d7fb43b0ad2ac90"}
Jul 28 19:46:41.775018 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:41.774981    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qljrk" event={"ID":"eb0afac4-b83a-4b2a-9942-35a729592390","Type":"ContainerStarted","Data":"31e153b79d3607d756a5cdde1de91c3afa0c070fe08d65da0329a000429d32f1"}
Jul 28 19:46:42.645809 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:42.645230    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:42.645809 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:42.645409    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:42.792892 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:42.792856    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" event={"ID":"cabbd4b8e7f726e6a0470a46a65225fe","Type":"ContainerStarted","Data":"fe1bbc072fef57bfa3797edcb7504c0738e8210f98fcf9111ffe757bf5c59ff9"}
Jul 28 19:46:43.113599 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:43.113386    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:43.113599 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.113571    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:43.113847 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.113650    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:46:47.11362615 +0000 UTC m=+10.013842978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:43.214998 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:43.214936    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:43.215176 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:43.215033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:43.215176 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.215098    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:43.215176 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.215165    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:46:43.215176 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.215171    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:47.215150315 +0000 UTC m=+10.115367151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:43.215176 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.215181    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:46:43.215511 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.215194    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:43.215511 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.215229    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:47.215218104 +0000 UTC m=+10.115434941 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:43.648313 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:43.648256    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:43.648899 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.648433    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:43.648899 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:43.648881    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:43.649077 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:43.648992    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:44.645659 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:44.645617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:44.645873 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:44.645767    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:45.646506 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:45.646469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:45.646506 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:45.646491    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:45.647096 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:45.646620    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:45.647096 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:45.646768    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:46.645391 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:46.645132    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:46.645391 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:46.645282    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:47.146713 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:47.146672    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:47.147209 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.146946    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:47.147209 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.147026    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:46:55.147000837 +0000 UTC m=+18.047217662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:47.248051    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:47.248138    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.248288    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.248378    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:55.248354382 +0000 UTC m=+18.148571218 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.248583    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.248608    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.248622    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:47.248741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.248668    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:46:55.248654072 +0000 UTC m=+18.148870913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:47.648095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:47.648060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:47.648258 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.648213    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:47.649588 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:47.649557    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:47.649745 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:47.649668    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:48.645613 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:48.645574    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:48.646077 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:48.645724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:49.645976 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:49.645435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:49.645976 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:49.645589    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:49.646802 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:49.646643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:49.646802 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:49.646763    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:50.645046 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:50.645010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:50.646383 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:50.645495    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:51.645989 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:51.645950    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:51.646186 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:51.646079    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:51.646486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:51.645953    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:51.646486 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:51.646450    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:52.645415 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:52.645332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:52.645579 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:52.645463    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:53.644989 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:53.644956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:53.644989 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:53.644987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:53.645504 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:53.645085    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:53.645504 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:53.645224    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:54.645529 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:54.645492    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:54.646039 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:54.645666    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:55.212347 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:55.212303    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:55.212541 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.212454    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:55.212541 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.212525    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:47:11.21250215 +0000 UTC m=+34.112718971 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:46:55.312812 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:55.312781    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:55.313039 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:55.312850    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:55.313233 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.313208    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:46:55.313331 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.313241    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:46:55.313331 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.313256    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:55.313331 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.313313    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:55.313462 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.313343    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:11.313323366 +0000 UTC m=+34.213540190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:46:55.313462 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.313399    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:11.313378897 +0000 UTC m=+34.213595790 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:46:55.645791 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:55.645707    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:55.646161 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:55.645703    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:55.646161 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.645856    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:55.646161 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:55.645933    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:56.645021 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:56.644986    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:56.645212 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:56.645111    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:57.645720 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:57.645688    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:57.646151 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:57.645814    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:46:57.646151 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:57.645865    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:57.646151 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:57.645968    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:58.644973 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:58.644935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:46:58.645149 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:58.645077    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:46:59.645516 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:59.645477    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:46:59.645949 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:46:59.645474    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:46:59.645949 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:59.645623    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:46:59.645949 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:46:59.645729    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:00.645475 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.645161    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:00.645598 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:00.645570    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:00.833019 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.832987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-phkfk" event={"ID":"9eb2af8b-a861-4a81-875e-e4ffb9f7bb94","Type":"ContainerStarted","Data":"19da3e8cd50f98c33fc04763f5fefcc56e6a6778f18065379ad13ee9b57d5467"}
Jul 28 19:47:00.834368 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.834336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hh7c9" event={"ID":"41ca7edc-a640-4fa8-97db-f7b908e53ed1","Type":"ContainerStarted","Data":"956fc8475d07c1cb4983afa16907bf534aa484d66d03e95b601ad12c30ae20a9"}
Jul 28 19:47:00.835630 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.835609    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xx69x" event={"ID":"e77e8b7e-663f-417e-8e2c-73683e752699","Type":"ContainerStarted","Data":"b9fffc41c986d2fa207c4be59285c549f3e2b2051e528d2794ab875762045cd0"}
Jul 28 19:47:00.836819 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.836797    2578 generic.go:356] "Generic (PLEG): container finished" podID="89149230-3f6c-41e2-a670-626669e37bd9" containerID="145214cc63370cb4ac5537cd9568af24cf8dd59177baad3873d15beeb5bcc584" exitCode=0
Jul 28 19:47:00.836924 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.836873    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerDied","Data":"145214cc63370cb4ac5537cd9568af24cf8dd59177baad3873d15beeb5bcc584"}
Jul 28 19:47:00.839470 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.839451    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"4138b2ee7406c7eead98e5e54daef4d68dc9edba4e02abcef5919b55147897f1"}
Jul 28 19:47:00.839563 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.839503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"2084f6bbe47ccd292e5ca4bfff33d7a3cb74fc976998907d9fada2d4682ed355"}
Jul 28 19:47:00.839563 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.839539    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"f881f3a432be0a4313e79ed17f7eea2ed2aa1c8ecf621c3e02189fb87b03c9aa"}
Jul 28 19:47:00.839679 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.839569    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"6d7d6d433ea8d7b619c64ddad0d5b86949e7820f6609f91684898681a000a0cc"}
Jul 28 19:47:00.842511 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.842486    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-974jf" event={"ID":"986541f9-ad8a-4755-b6bc-37e010abdbdb","Type":"ContainerStarted","Data":"8bf6e49537714178b42cfeca1414fbfcecc172496931e1119daa2031d78ff5de"}
Jul 28 19:47:00.844050 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.844025    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-qljrk" event={"ID":"eb0afac4-b83a-4b2a-9942-35a729592390","Type":"ContainerStarted","Data":"535dd63b677135ba4b02f894554f56a5cfb9ef6b5816aa847020a77279680ef7"}
Jul 28 19:47:00.845663 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.845645    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v" event={"ID":"59603633-ac5f-4664-b137-310733f393f9","Type":"ContainerStarted","Data":"f87fff4468bd1ee79196059e586477302b27225f266958c2a556b2e7ef0c1219"}
Jul 28 19:47:00.853849 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.853807    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-phkfk" podStartSLOduration=3.439118888 podStartE2EDuration="22.853794802s" totalImagesPullingTime="19.414675914s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:00.85335841 +0000 UTC m=+23.753575252" watchObservedRunningTime="2026-07-28 19:47:00.853794802 +0000 UTC m=+23.754011644"
Jul 28 19:47:00.853959 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.853926    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-160.ec2.internal" podStartSLOduration=21.85392147 podStartE2EDuration="21.85392147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:46:42.809095086 +0000 UTC m=+5.709311950" watchObservedRunningTime="2026-07-28 19:47:00.85392147 +0000 UTC m=+23.754138312"
Jul 28 19:47:00.891263 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.891227    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xx69x" podStartSLOduration=3.489619968 podStartE2EDuration="22.891211854s" totalImagesPullingTime="19.401591886s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:00.89080718 +0000 UTC m=+23.791024022" watchObservedRunningTime="2026-07-28 19:47:00.891211854 +0000 UTC m=+23.791428697"
Jul 28 19:47:00.918059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.918017    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-qljrk" podStartSLOduration=3.434678277 podStartE2EDuration="22.918002882s" totalImagesPullingTime="19.483324605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:00.91748956 +0000 UTC m=+23.817706402" watchObservedRunningTime="2026-07-28 19:47:00.918002882 +0000 UTC m=+23.818219720"
Jul 28 19:47:00.930049 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.930008    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-974jf" podStartSLOduration=2.513036317 podStartE2EDuration="21.929998009s" totalImagesPullingTime="19.416961692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:00.929701978 +0000 UTC m=+23.829918820" watchObservedRunningTime="2026-07-28 19:47:00.929998009 +0000 UTC m=+23.830214850"
Jul 28 19:47:00.949764 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:00.949718    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hh7c9" podStartSLOduration=2.547957116 podStartE2EDuration="21.949703753s" totalImagesPullingTime="19.401746637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:00.949465231 +0000 UTC m=+23.849682073" watchObservedRunningTime="2026-07-28 19:47:00.949703753 +0000 UTC m=+23.849920594"
Jul 28 19:47:01.645051 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.645009    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:01.645240 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:01.645138    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:01.645240 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.645193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:01.645373 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:01.645341    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:01.844343 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.844319    2578 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 19:47:01.851997 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.851959    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"e7fde34d52aece74730b856ede056d1dd918ba7f8eea4aba3f104966a98faf15"}
Jul 28 19:47:01.852106 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.852051    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"ea2b831f8d0e6c14dbc70d8ee592e5daa177186c7b1cb5c9ec09bacbc0501cf8"}
Jul 28 19:47:01.853702 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.853663    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxx52" event={"ID":"a37fb47a-b3ea-40cf-90e2-28e48e50a234","Type":"ContainerStarted","Data":"f623c7d51e7f4895e272ad61afe024b0f4bbf1676927a43e669aacac75a59869"}
Jul 28 19:47:01.856297 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.855608    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v" event={"ID":"59603633-ac5f-4664-b137-310733f393f9","Type":"ContainerStarted","Data":"83a3a714021529c88d1bd5fb75dafac5b2200dfac7b905020456ac1dbc993cfa"}
Jul 28 19:47:01.868028 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:01.867984    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zxx52" podStartSLOduration=4.443899592 podStartE2EDuration="23.867970374s" totalImagesPullingTime="19.424070782s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:01.867402449 +0000 UTC m=+24.767619290" watchObservedRunningTime="2026-07-28 19:47:01.867970374 +0000 UTC m=+24.768187215"
Jul 28 19:47:02.556662 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.556518    2578 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T19:47:01.844336744Z","UUID":"abfe66e1-e10d-44c3-95b3-abc6988a28f9","Handler":null,"Name":"","Endpoint":""}
Jul 28 19:47:02.558156 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.558135    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 19:47:02.558332 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.558171    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 19:47:02.645757 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.645727    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:02.645929 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:02.645844    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:02.772959 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.772919    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:47:02.773693 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.773669    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:47:02.858106 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.857962    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:47:02.858480 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:02.858356    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hh7c9"
Jul 28 19:47:03.645018 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:03.644983    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:03.645253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:03.644984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:03.645253 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:03.645124    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:03.645253 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:03.645193    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:03.865080 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:03.865045    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"016e5f3b0bce593f53e73620f0697ca79cf1471b86c4df3231a68caca8033fca"}
Jul 28 19:47:03.867384 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:03.867353    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v" event={"ID":"59603633-ac5f-4664-b137-310733f393f9","Type":"ContainerStarted","Data":"b3bdb49a3459beb7f1320f0e8b9f29d51c92123e46d98bab29e2f6751a45521c"}
Jul 28 19:47:03.885482 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:03.885418    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8nq4v" podStartSLOduration=4.216451595 podStartE2EDuration="25.88540181s" totalImagesPullingTime="21.668950215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:03.885048227 +0000 UTC m=+26.785265070" watchObservedRunningTime="2026-07-28 19:47:03.88540181 +0000 UTC m=+26.785618651"
Jul 28 19:47:04.645843 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:04.645643    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:04.646033 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:04.645949    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:05.645410 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.645185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:05.646313 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.645215    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:05.646313 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:05.645516    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:05.646313 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:05.645549    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:05.873553 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.873514    2578 generic.go:356] "Generic (PLEG): container finished" podID="89149230-3f6c-41e2-a670-626669e37bd9" containerID="5f483798d5969f7904a5f058a83745e2e6807ac767d386447ef1fde7ef28c700" exitCode=0
Jul 28 19:47:05.873728 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.873601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerDied","Data":"5f483798d5969f7904a5f058a83745e2e6807ac767d386447ef1fde7ef28c700"}
Jul 28 19:47:05.877606 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.877580    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" event={"ID":"e4f7328e-2111-43f3-a7bc-f47113f1c2a5","Type":"ContainerStarted","Data":"68b129f0cd753a0fcc0a509c530a1b8574f8b766594ee3aa74159ef987eec611"}
Jul 28 19:47:05.877850 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.877833    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:47:05.877942 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.877896    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:47:05.878002 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.877945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:47:05.894506 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.894479    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:47:05.894642 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.894586    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:47:05.923783 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:05.923687    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w" podStartSLOduration=8.423359911 podStartE2EDuration="27.923672302s" totalImagesPullingTime="19.500312391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:05.922553836 +0000 UTC m=+28.822770678" watchObservedRunningTime="2026-07-28 19:47:05.923672302 +0000 UTC m=+28.823889143"
Jul 28 19:47:06.645474 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:06.645435    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:06.645984 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:06.645585    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:07.185364 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.185333    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lt7fb"]
Jul 28 19:47:07.185493 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.185460    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:07.185561 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:07.185545    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:07.188731 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.188696    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n995h"]
Jul 28 19:47:07.190722 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.190624    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m9tq9"]
Jul 28 19:47:07.190824 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.190798    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:07.190964 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:07.190940    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:07.192870 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.192831    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:07.193453 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:07.193422    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:07.884104 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.884069    2578 generic.go:356] "Generic (PLEG): container finished" podID="89149230-3f6c-41e2-a670-626669e37bd9" containerID="17870a7ef10ece3d5debb76404daafa538dcd9bde07e026e1b79522f43fa4dfd" exitCode=0
Jul 28 19:47:07.884614 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:07.884161    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerDied","Data":"17870a7ef10ece3d5debb76404daafa538dcd9bde07e026e1b79522f43fa4dfd"}
Jul 28 19:47:08.645287 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:08.645228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:08.645287 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:08.645249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:08.645509 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:08.645254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:08.645509 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:08.645370    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:08.645509 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:08.645440    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:08.645637 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:08.645538    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:09.890329 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:09.890137    2578 generic.go:356] "Generic (PLEG): container finished" podID="89149230-3f6c-41e2-a670-626669e37bd9" containerID="06d606526daa6f0958d538adf60b4bb90ae5cd935cbfc9131ff04b2b7b2864a8" exitCode=0
Jul 28 19:47:09.890717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:09.890216    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerDied","Data":"06d606526daa6f0958d538adf60b4bb90ae5cd935cbfc9131ff04b2b7b2864a8"}
Jul 28 19:47:10.645310 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:10.645277    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:10.645310 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:10.645295    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:10.645554 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:10.645282    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:10.645554 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:10.645412    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:10.645554 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:10.645480    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:10.645702 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:10.645564    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:11.239237 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:11.239197    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:11.239711 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.239346    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:47:11.239711 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.239409    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs podName:03c8c9fd-d068-468d-a118-e2a49e136cee nodeName:}" failed. No retries permitted until 2026-07-28 19:47:43.239394715 +0000 UTC m=+66.139611538 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs") pod "network-metrics-daemon-m9tq9" (UID: "03c8c9fd-d068-468d-a118-e2a49e136cee") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 19:47:11.340514 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:11.340263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:11.340514 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:11.340509    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:11.340739 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.340450    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 19:47:11.340739 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.340565    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 19:47:11.340739 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.340579    2578 projected.go:196] Error preparing data for projected volume kube-api-access-ps65r for pod openshift-network-diagnostics/network-check-target-n995h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:47:11.340739 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.340624    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 19:47:11.340739 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.340645    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r podName:86914da0-7e38-465d-bf4b-1207416887a2 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:43.340626618 +0000 UTC m=+66.240843458 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ps65r" (UniqueName: "kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r") pod "network-check-target-n995h" (UID: "86914da0-7e38-465d-bf4b-1207416887a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 19:47:11.340739 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:11.340678    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret podName:4cad5271-713f-4162-b9ad-00cdc9dd0221 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:43.340661908 +0000 UTC m=+66.240878731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret") pod "global-pull-secret-syncer-lt7fb" (UID: "4cad5271-713f-4162-b9ad-00cdc9dd0221") : object "kube-system"/"original-pull-secret" not registered
Jul 28 19:47:12.645235 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.645197    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:12.645828 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.645243    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:12.645828 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.645248    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:12.645828 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:12.645345    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-n995h" podUID="86914da0-7e38-465d-bf4b-1207416887a2"
Jul 28 19:47:12.645828 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:12.645439    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m9tq9" podUID="03c8c9fd-d068-468d-a118-e2a49e136cee"
Jul 28 19:47:12.645828 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:12.645536    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lt7fb" podUID="4cad5271-713f-4162-b9ad-00cdc9dd0221"
Jul 28 19:47:12.891357 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.891326    2578 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-160.ec2.internal" event="NodeReady"
Jul 28 19:47:12.891528 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.891500    2578 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 19:47:12.924355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.924085    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-698cf84dbb-dcxv5"]
Jul 28 19:47:12.926548 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.926516    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:12.929205 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.929158    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 19:47:12.929205 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.929167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 19:47:12.929615 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.929592    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 19:47:12.930113 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.929781    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c54t5\""
Jul 28 19:47:12.935764 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.935736    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jh7xz"]
Jul 28 19:47:12.938793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.938777    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 19:47:12.939066 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.939036    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6v2wp"]
Jul 28 19:47:12.939187 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.939171    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:12.940984 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.940963    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:12.941549 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.941531    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wfpzj\""
Jul 28 19:47:12.942024 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.941996    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-698cf84dbb-dcxv5"]
Jul 28 19:47:12.942252 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.942233    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 19:47:12.942557 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.942517    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 19:47:12.943779 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.943762    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 19:47:12.943990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.943869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zg9s9\""
Jul 28 19:47:12.943990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.943925    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 19:47:12.943990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.943976    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 19:47:12.950370 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.950328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jh7xz"]
Jul 28 19:47:12.950717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:12.950700    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6v2wp"]
Jul 28 19:47:13.054494 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054457    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-image-registry-private-configuration\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054494 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054498    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-registry-certificates\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054522    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.054712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-installation-pull-secrets\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054606    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-config-volume\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.054712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054669    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054712 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-trusted-ca\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9zff\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-kube-api-access-l9zff\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-tmp-dir\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054788    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33895c8b-396f-428a-98ae-edb4ce1cf0c4-cert\") pod \"ingress-canary-6v2wp\" (UID: \"33895c8b-396f-428a-98ae-edb4ce1cf0c4\") " pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054847    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34065413-44cf-4622-9f6f-be947f1d7c82-ca-trust-extracted\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8qwq\" (UniqueName: \"kubernetes.io/projected/33895c8b-396f-428a-98ae-edb4ce1cf0c4-kube-api-access-k8qwq\") pod \"ingress-canary-6v2wp\" (UID: \"33895c8b-396f-428a-98ae-edb4ce1cf0c4\") " pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-bound-sa-token\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.054966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.054937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqt6\" (UniqueName: \"kubernetes.io/projected/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-kube-api-access-brqt6\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.071450 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.071410    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:13.154389 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.154360    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:13.156185 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-tmp-dir\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.156325 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156189    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33895c8b-396f-428a-98ae-edb4ce1cf0c4-cert\") pod \"ingress-canary-6v2wp\" (UID: \"33895c8b-396f-428a-98ae-edb4ce1cf0c4\") " pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.156325 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156220    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34065413-44cf-4622-9f6f-be947f1d7c82-ca-trust-extracted\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156402 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8qwq\" (UniqueName: \"kubernetes.io/projected/33895c8b-396f-428a-98ae-edb4ce1cf0c4-kube-api-access-k8qwq\") pod \"ingress-canary-6v2wp\" (UID: \"33895c8b-396f-428a-98ae-edb4ce1cf0c4\") " pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.156402 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156381    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-bound-sa-token\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156525 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156501    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqt6\" (UniqueName: \"kubernetes.io/projected/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-kube-api-access-brqt6\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.156577 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156562    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-image-registry-private-configuration\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156627 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156578    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-tmp-dir\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.156627 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156591    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-registry-certificates\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156667    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-installation-pull-secrets\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156674    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34065413-44cf-4622-9f6f-be947f1d7c82-ca-trust-extracted\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-config-volume\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156747    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-trusted-ca\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.156762    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9zff\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-kube-api-access-l9zff\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.156773 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.156766    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:47:13.157183 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.156817    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls podName:fd16eed2-e3aa-4672-bdfc-c9c10b376e9b nodeName:}" failed. No retries permitted until 2026-07-28 19:47:13.656800371 +0000 UTC m=+36.557017194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls") pod "dns-default-jh7xz" (UID: "fd16eed2-e3aa-4672-bdfc-c9c10b376e9b") : secret "dns-default-metrics-tls" not found
Jul 28 19:47:13.157183 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.157045    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:47:13.157183 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.157061    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf84dbb-dcxv5: secret "image-registry-tls" not found
Jul 28 19:47:13.157183 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.157135    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls podName:34065413-44cf-4622-9f6f-be947f1d7c82 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:13.657109285 +0000 UTC m=+36.557326107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls") pod "image-registry-698cf84dbb-dcxv5" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82") : secret "image-registry-tls" not found
Jul 28 19:47:13.157183 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.157159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-registry-certificates\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.157480 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.157459    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-config-volume\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.157828 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.157801    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-trusted-ca\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.161097 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.161075    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-installation-pull-secrets\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.161186 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.161087    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/33895c8b-396f-428a-98ae-edb4ce1cf0c4-cert\") pod \"ingress-canary-6v2wp\" (UID: \"33895c8b-396f-428a-98ae-edb4ce1cf0c4\") " pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.161683 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.161657    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-image-registry-private-configuration\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.165281 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.165239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9zff\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-kube-api-access-l9zff\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.165620 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.165598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8qwq\" (UniqueName: \"kubernetes.io/projected/33895c8b-396f-428a-98ae-edb4ce1cf0c4-kube-api-access-k8qwq\") pod \"ingress-canary-6v2wp\" (UID: \"33895c8b-396f-428a-98ae-edb4ce1cf0c4\") " pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.165702 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.165688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-bound-sa-token\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.165870 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.165846    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqt6\" (UniqueName: \"kubernetes.io/projected/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-kube-api-access-brqt6\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.256284 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.256185    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6v2wp"
Jul 28 19:47:13.478197 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.478163    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:13.534062 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.533962    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:13.660899 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.660861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:13.661647 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:13.660917    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:13.661647 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.661037    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:47:13.661647 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.661114    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls podName:fd16eed2-e3aa-4672-bdfc-c9c10b376e9b nodeName:}" failed. No retries permitted until 2026-07-28 19:47:14.661093251 +0000 UTC m=+37.561310073 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls") pod "dns-default-jh7xz" (UID: "fd16eed2-e3aa-4672-bdfc-c9c10b376e9b") : secret "dns-default-metrics-tls" not found
Jul 28 19:47:13.661647 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.661038    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:47:13.661647 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.661148    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf84dbb-dcxv5: secret "image-registry-tls" not found
Jul 28 19:47:13.661647 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:13.661209    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls podName:34065413-44cf-4622-9f6f-be947f1d7c82 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:14.661196963 +0000 UTC m=+37.561413799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls") pod "image-registry-698cf84dbb-dcxv5" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82") : secret "image-registry-tls" not found
Jul 28 19:47:14.214800 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.214771    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6v2wp"]
Jul 28 19:47:14.219322 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:14.219297    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33895c8b_396f_428a_98ae_edb4ce1cf0c4.slice/crio-0470a05896c5b36bd4454b7f9d65f129cf16d27edc6f470d4d07dfd99a5f3a01 WatchSource:0}: Error finding container 0470a05896c5b36bd4454b7f9d65f129cf16d27edc6f470d4d07dfd99a5f3a01: Status 404 returned error can't find the container with id 0470a05896c5b36bd4454b7f9d65f129cf16d27edc6f470d4d07dfd99a5f3a01
Jul 28 19:47:14.430857 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.430748    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:14.645125 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.645092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:14.645336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.645092    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:14.645336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.645209    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:14.649103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.649076    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 19:47:14.649231 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.649114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 19:47:14.649231 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.649114    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lhtrs\""
Jul 28 19:47:14.649231 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.649181    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 19:47:14.649231 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.649192    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4wgt\""
Jul 28 19:47:14.649231 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.649113    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 19:47:14.670716 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.670683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:14.671135 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.670728    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:14.671135 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:14.670818    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:47:14.671135 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:14.670828    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf84dbb-dcxv5: secret "image-registry-tls" not found
Jul 28 19:47:14.671135 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:14.670855    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:47:14.671135 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:14.670874    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls podName:34065413-44cf-4622-9f6f-be947f1d7c82 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:16.670861064 +0000 UTC m=+39.571077897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls") pod "image-registry-698cf84dbb-dcxv5" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82") : secret "image-registry-tls" not found
Jul 28 19:47:14.671135 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:14.670913    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls podName:fd16eed2-e3aa-4672-bdfc-c9c10b376e9b nodeName:}" failed. No retries permitted until 2026-07-28 19:47:16.670893096 +0000 UTC m=+39.571109935 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls") pod "dns-default-jh7xz" (UID: "fd16eed2-e3aa-4672-bdfc-c9c10b376e9b") : secret "dns-default-metrics-tls" not found
Jul 28 19:47:14.904955 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.904869    2578 generic.go:356] "Generic (PLEG): container finished" podID="89149230-3f6c-41e2-a670-626669e37bd9" containerID="0e852b7dc262461c161d3926f80502d171e037de4ee61a22c3df70af02a8e175" exitCode=0
Jul 28 19:47:14.905098 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.904957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerDied","Data":"0e852b7dc262461c161d3926f80502d171e037de4ee61a22c3df70af02a8e175"}
Jul 28 19:47:14.906111 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:14.906092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6v2wp" event={"ID":"33895c8b-396f-428a-98ae-edb4ce1cf0c4","Type":"ContainerStarted","Data":"0470a05896c5b36bd4454b7f9d65f129cf16d27edc6f470d4d07dfd99a5f3a01"}
Jul 28 19:47:15.783480 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:15.783438    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:15.839431 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:15.839344    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:15.912004 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:15.911973    2578 generic.go:356] "Generic (PLEG): container finished" podID="89149230-3f6c-41e2-a670-626669e37bd9" containerID="eaad00429dcdd76720889120b1e5446e82a716d1a14217bd32bb2b350ad54c6a" exitCode=0
Jul 28 19:47:15.912175 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:15.912019    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerDied","Data":"eaad00429dcdd76720889120b1e5446e82a716d1a14217bd32bb2b350ad54c6a"}
Jul 28 19:47:16.690068 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:16.690035    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:16.690253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:16.690094    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:16.690253 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:16.690217    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:47:16.690394 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:16.690287    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:47:16.690394 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:16.690303    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf84dbb-dcxv5: secret "image-registry-tls" not found
Jul 28 19:47:16.690394 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:16.690313    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls podName:fd16eed2-e3aa-4672-bdfc-c9c10b376e9b nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.690291148 +0000 UTC m=+43.590507984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls") pod "dns-default-jh7xz" (UID: "fd16eed2-e3aa-4672-bdfc-c9c10b376e9b") : secret "dns-default-metrics-tls" not found
Jul 28 19:47:16.690394 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:16.690352    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls podName:34065413-44cf-4622-9f6f-be947f1d7c82 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.690336055 +0000 UTC m=+43.590552890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls") pod "image-registry-698cf84dbb-dcxv5" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82") : secret "image-registry-tls" not found
Jul 28 19:47:16.918741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:16.918699    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-knqbt" event={"ID":"89149230-3f6c-41e2-a670-626669e37bd9","Type":"ContainerStarted","Data":"2f4987c14602bbfbf139445fb2eb4fdd2bc20076880b16bbe64c88fc5c26f78c"}
Jul 28 19:47:16.943510 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:16.941340    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-knqbt" podStartSLOduration=8.687970742 podStartE2EDuration="38.941315867s" totalImagesPullingTime="30.253345125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:16.940778381 +0000 UTC m=+39.840995225" watchObservedRunningTime="2026-07-28 19:47:16.941315867 +0000 UTC m=+39.841532710"
Jul 28 19:47:17.059032 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:17.058991    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:17.113707 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:17.113669    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:17.403072 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:17.402870    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:17.756014 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:17.755983    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:17.922473 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:17.922426    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6v2wp" event={"ID":"33895c8b-396f-428a-98ae-edb4ce1cf0c4","Type":"ContainerStarted","Data":"19fbc3bf38851ba00252219bc5abfadec5e741fe2a6d637e78aeffd528d30564"}
Jul 28 19:47:17.937370 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:17.937323    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6v2wp" podStartSLOduration=3.013961728 podStartE2EDuration="5.937307926s" totalImagesPullingTime="2.923346198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:17.93702989 +0000 UTC m=+40.837246732" watchObservedRunningTime="2026-07-28 19:47:17.937307926 +0000 UTC m=+40.837524769"
Jul 28 19:47:18.888958 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.888925    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"]
Jul 28 19:47:18.909715 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.909676    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"]
Jul 28 19:47:18.910807 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.910781    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:18.916501 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.916471    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 19:47:18.916763 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.916712    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-djnv2\""
Jul 28 19:47:18.916950 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.916931    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:47:18.917197 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.917178    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 19:47:18.917394 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.917376    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 19:47:18.918786 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.918756    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b522g\" (UniqueName: \"kubernetes.io/projected/9d48cb85-c537-4a11-90f7-1da5815cc0ba-kube-api-access-b522g\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:18.918895 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.918837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d48cb85-c537-4a11-90f7-1da5815cc0ba-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:18.918942 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.918908    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d48cb85-c537-4a11-90f7-1da5815cc0ba-config\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:18.920482 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.920456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-nnfhr"]
Jul 28 19:47:18.924747 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.924714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"
Jul 28 19:47:18.937919 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.937887    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 19:47:18.938184 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.938134    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-z8grj\""
Jul 28 19:47:18.938283 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.937987    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:47:18.938990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.938947    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"]
Jul 28 19:47:18.939099 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.939075    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:18.942345 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.942302    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:47:18.942803 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.942513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 19:47:18.942803 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.942513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 19:47:18.942803 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.942566    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dbwrw\""
Jul 28 19:47:18.942803 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.942581    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 19:47:18.947466 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.947445    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 19:47:18.949165 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.949148    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-86b574f647-594xn"]
Jul 28 19:47:18.949291 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.949249    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:18.952117 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.952098    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 19:47:18.952206 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.952171    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:47:18.952289 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.952218    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-vnlt2\""
Jul 28 19:47:18.952289 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.952247    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 19:47:18.952402 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.952312    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 19:47:18.963230 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.963209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"]
Jul 28 19:47:18.963327 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.963236    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"]
Jul 28 19:47:18.963327 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.963293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"]
Jul 28 19:47:18.963327 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.963309    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86b574f647-594xn"]
Jul 28 19:47:18.963327 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.963325    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-nnfhr"]
Jul 28 19:47:18.963452 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.963333    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:18.965702 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.965685    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 19:47:18.965808 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.965729    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 19:47:18.965882 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.965864    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 19:47:18.965961 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.965948    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 19:47:18.966017 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.965971    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 19:47:18.966370 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.966354    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 19:47:18.966467 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.966368    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ssps7\""
Jul 28 19:47:18.993550 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:18.993522    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"]
Jul 28 19:47:19.009710 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.009576    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"]
Jul 28 19:47:19.009845 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.009796    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.015482 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.015460    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 19:47:19.015864 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.015839    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 19:47:19.015968 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.015876    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nfftp\""
Jul 28 19:47:19.019387 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019369    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0377c7ec-588d-42a2-aed1-e21b9f626411-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.019486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019405    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cmq5\" (UniqueName: \"kubernetes.io/projected/6cb891fa-f2eb-429d-abcd-ee896160e98a-kube-api-access-7cmq5\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.019486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019424    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-default-certificate\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.019486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019445    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.019605 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a270ba75-d93d-4d05-ab56-523a14f234a9-serving-cert\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.019605 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a270ba75-d93d-4d05-ab56-523a14f234a9-trusted-ca\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.019605 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019585    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d48cb85-c537-4a11-90f7-1da5815cc0ba-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.019698 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.019698 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ad0030e8-c33f-49c3-bb42-d0b809f19f98-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.019765 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019714    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.019765 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfncz\" (UniqueName: \"kubernetes.io/projected/3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd-kube-api-access-gfncz\") pod \"volume-data-source-validator-549c6d54cf-cfccp\" (UID: \"3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"
Jul 28 19:47:19.019834 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019765    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a270ba75-d93d-4d05-ab56-523a14f234a9-config\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.019834 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019817    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm5k2\" (UniqueName: \"kubernetes.io/projected/a270ba75-d93d-4d05-ab56-523a14f234a9-kube-api-access-mm5k2\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.019834 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019832    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-stats-auth\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.019940 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d48cb85-c537-4a11-90f7-1da5815cc0ba-config\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.019940 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.019884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b522g\" (UniqueName: \"kubernetes.io/projected/9d48cb85-c537-4a11-90f7-1da5815cc0ba-kube-api-access-b522g\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.020069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.020048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0377c7ec-588d-42a2-aed1-e21b9f626411-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.020145 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.020125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpg59\" (UniqueName: \"kubernetes.io/projected/0377c7ec-588d-42a2-aed1-e21b9f626411-kube-api-access-dpg59\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.020475 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.020458    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9d48cb85-c537-4a11-90f7-1da5815cc0ba-config\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.023560 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.023543    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9d48cb85-c537-4a11-90f7-1da5815cc0ba-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.023637 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.023552    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"]
Jul 28 19:47:19.023637 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.023615    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.026847 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.026825    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5lnhb\""
Jul 28 19:47:19.027169 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.026853    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:47:19.027533 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.026853    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 19:47:19.027882 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.026885    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 19:47:19.035109 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.035088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b522g\" (UniqueName: \"kubernetes.io/projected/9d48cb85-c537-4a11-90f7-1da5815cc0ba-kube-api-access-b522g\") pod \"service-ca-operator-5d6c75f84f-lbkmf\" (UID: \"9d48cb85-c537-4a11-90f7-1da5815cc0ba\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.044002 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.043978    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-j66nk"]
Jul 28 19:47:19.044124 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.044090    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.046846 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.046827    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-52l28\""
Jul 28 19:47:19.047826 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.047802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 19:47:19.047826 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.047804    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 19:47:19.048083 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.048069    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 19:47:19.048210 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.048076    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 19:47:19.048762 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.048734    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:19.067870 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.067851    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"]
Jul 28 19:47:19.067959 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.067879    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"]
Jul 28 19:47:19.067959 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.067894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"]
Jul 28 19:47:19.067959 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.067919    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-j66nk"]
Jul 28 19:47:19.067959 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.067946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.070559 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.070539    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 19:47:19.070559 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.070554    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 19:47:19.070724 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.070606    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-b8t9k\""
Jul 28 19:47:19.070724 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.070605    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 19:47:19.070724 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.070554    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 19:47:19.075727 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.075706    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 19:47:19.089818 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.089791    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"]
Jul 28 19:47:19.109397 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.109373    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:19.114386 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.114354    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"]
Jul 28 19:47:19.114472 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.114438    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"
Jul 28 19:47:19.117455 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.117436    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-5gkp7\""
Jul 28 19:47:19.120752 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plbf2\" (UniqueName: \"kubernetes.io/projected/3adb22c5-5525-40b6-a40c-42a13c6abd3a-kube-api-access-plbf2\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.120855 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120772    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3adb22c5-5525-40b6-a40c-42a13c6abd3a-snapshots\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.120855 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120800    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb22c5-5525-40b6-a40c-42a13c6abd3a-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.120855 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120846    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0377c7ec-588d-42a2-aed1-e21b9f626411-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.120997 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120864    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3adb22c5-5525-40b6-a40c-42a13c6abd3a-tmp\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.120997 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120963    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg59\" (UniqueName: \"kubernetes.io/projected/0377c7ec-588d-42a2-aed1-e21b9f626411-kube-api-access-dpg59\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.121102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.120995    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3adb22c5-5525-40b6-a40c-42a13c6abd3a-serving-cert\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.121102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0377c7ec-588d-42a2-aed1-e21b9f626411-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.121102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121050    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cmq5\" (UniqueName: \"kubernetes.io/projected/6cb891fa-f2eb-429d-abcd-ee896160e98a-kube-api-access-7cmq5\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.121102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-default-certificate\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.121102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121091    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121118    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121146    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a270ba75-d93d-4d05-ab56-523a14f234a9-serving-cert\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121169    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a270ba75-d93d-4d05-ab56-523a14f234a9-trusted-ca\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vdqn\" (UniqueName: \"kubernetes.io/projected/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-kube-api-access-4vdqn\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121261    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f985925d-7690-4f3d-b33d-d12804131a78-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121313    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ad0030e8-c33f-49c3-bb42-d0b809f19f98-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.121355 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121338    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.121355    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4slk\" (UniqueName: \"kubernetes.io/projected/f985925d-7690-4f3d-b33d-d12804131a78-kube-api-access-m4slk\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0377c7ec-588d-42a2-aed1-e21b9f626411-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121393    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb22c5-5525-40b6-a40c-42a13c6abd3a-service-ca-bundle\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.121426    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert podName:ad0030e8-c33f-49c3-bb42-d0b809f19f98 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:19.621405243 +0000 UTC m=+42.521622082 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-gsjpt" (UID: "ad0030e8-c33f-49c3-bb42-d0b809f19f98") : secret "networking-console-plugin-cert" not found
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121641    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfncz\" (UniqueName: \"kubernetes.io/projected/3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd-kube-api-access-gfncz\") pod \"volume-data-source-validator-549c6d54cf-cfccp\" (UID: \"3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"
Jul 28 19:47:19.121741 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121675    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a270ba75-d93d-4d05-ab56-523a14f234a9-config\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.122093 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121879    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8tl\" (UniqueName: \"kubernetes.io/projected/a98e6895-f9a4-43bd-8cdd-da4b44cc5c95-kube-api-access-4f8tl\") pod \"network-check-source-7d88cb55d8-jq4rd\" (UID: \"a98e6895-f9a4-43bd-8cdd-da4b44cc5c95\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"
Jul 28 19:47:19.122093 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121913    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm5k2\" (UniqueName: \"kubernetes.io/projected/a270ba75-d93d-4d05-ab56-523a14f234a9-kube-api-access-mm5k2\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.122093 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121939    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-stats-auth\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.122093 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.121967    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.122093 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.121995    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 19:47:19.122093 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.122054    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:19.622039068 +0000 UTC m=+42.522255906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : secret "router-metrics-certs-default" not found
Jul 28 19:47:19.122410 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.122138    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:19.622127083 +0000 UTC m=+42.522343942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : configmap references non-existent config key: service-ca.crt
Jul 28 19:47:19.122410 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.122256    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ad0030e8-c33f-49c3-bb42-d0b809f19f98-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.122410 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.122369    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a270ba75-d93d-4d05-ab56-523a14f234a9-trusted-ca\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.123770 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.123748    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0377c7ec-588d-42a2-aed1-e21b9f626411-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.124092 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.124076    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a270ba75-d93d-4d05-ab56-523a14f234a9-serving-cert\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.124170 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.124089    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-default-certificate\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.124604 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.124589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-stats-auth\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.132013 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.131987    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpg59\" (UniqueName: \"kubernetes.io/projected/0377c7ec-588d-42a2-aed1-e21b9f626411-kube-api-access-dpg59\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2\" (UID: \"0377c7ec-588d-42a2-aed1-e21b9f626411\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.132180 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.132160    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a270ba75-d93d-4d05-ab56-523a14f234a9-config\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.133019 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.132998    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cmq5\" (UniqueName: \"kubernetes.io/projected/6cb891fa-f2eb-429d-abcd-ee896160e98a-kube-api-access-7cmq5\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.133883 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.133861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfncz\" (UniqueName: \"kubernetes.io/projected/3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd-kube-api-access-gfncz\") pod \"volume-data-source-validator-549c6d54cf-cfccp\" (UID: \"3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"
Jul 28 19:47:19.134378 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.134363    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm5k2\" (UniqueName: \"kubernetes.io/projected/a270ba75-d93d-4d05-ab56-523a14f234a9-kube-api-access-mm5k2\") pod \"console-operator-584dcb6b54-nnfhr\" (UID: \"a270ba75-d93d-4d05-ab56-523a14f234a9\") " pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.163161 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.163128    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:19.222472 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222389    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.222472 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222453    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vdqn\" (UniqueName: \"kubernetes.io/projected/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-kube-api-access-4vdqn\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.222666 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222476    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f985925d-7690-4f3d-b33d-d12804131a78-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.222666 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222502    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4slk\" (UniqueName: \"kubernetes.io/projected/f985925d-7690-4f3d-b33d-d12804131a78-kube-api-access-m4slk\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.222666 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb22c5-5525-40b6-a40c-42a13c6abd3a-service-ca-bundle\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.222666 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.222542    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 19:47:19.222666 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222544    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8tl\" (UniqueName: \"kubernetes.io/projected/a98e6895-f9a4-43bd-8cdd-da4b44cc5c95-kube-api-access-4f8tl\") pod \"network-check-source-7d88cb55d8-jq4rd\" (UID: \"a98e6895-f9a4-43bd-8cdd-da4b44cc5c95\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"
Jul 28 19:47:19.222666 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.222659    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls podName:9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:19.722634928 +0000 UTC m=+42.622851768 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wq9qq" (UID: "9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6") : secret "samples-operator-tls" not found
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222709    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plbf2\" (UniqueName: \"kubernetes.io/projected/3adb22c5-5525-40b6-a40c-42a13c6abd3a-kube-api-access-plbf2\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3adb22c5-5525-40b6-a40c-42a13c6abd3a-snapshots\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb22c5-5525-40b6-a40c-42a13c6abd3a-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.222835    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222870    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3adb22c5-5525-40b6-a40c-42a13c6abd3a-tmp\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.222930 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.222928    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls podName:f985925d-7690-4f3d-b33d-d12804131a78 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:19.722909771 +0000 UTC m=+42.623126605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j8m4d" (UID: "f985925d-7690-4f3d-b33d-d12804131a78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:19.223228 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.222990    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3adb22c5-5525-40b6-a40c-42a13c6abd3a-serving-cert\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.223228 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.223128    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb22c5-5525-40b6-a40c-42a13c6abd3a-service-ca-bundle\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.223401 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.223378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f985925d-7690-4f3d-b33d-d12804131a78-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.223628 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.223611    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb22c5-5525-40b6-a40c-42a13c6abd3a-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.229984 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.229966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3adb22c5-5525-40b6-a40c-42a13c6abd3a-tmp\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.230060 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.230047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3adb22c5-5525-40b6-a40c-42a13c6abd3a-snapshots\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.230115 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.230083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3adb22c5-5525-40b6-a40c-42a13c6abd3a-serving-cert\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.233377 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.233355    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4slk\" (UniqueName: \"kubernetes.io/projected/f985925d-7690-4f3d-b33d-d12804131a78-kube-api-access-m4slk\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.233489 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.233469    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vdqn\" (UniqueName: \"kubernetes.io/projected/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-kube-api-access-4vdqn\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.233525 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.233485    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-plbf2\" (UniqueName: \"kubernetes.io/projected/3adb22c5-5525-40b6-a40c-42a13c6abd3a-kube-api-access-plbf2\") pod \"insights-operator-5885fdb759-j66nk\" (UID: \"3adb22c5-5525-40b6-a40c-42a13c6abd3a\") " pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.233525 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.233472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8tl\" (UniqueName: \"kubernetes.io/projected/a98e6895-f9a4-43bd-8cdd-da4b44cc5c95-kube-api-access-4f8tl\") pod \"network-check-source-7d88cb55d8-jq4rd\" (UID: \"a98e6895-f9a4-43bd-8cdd-da4b44cc5c95\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"
Jul 28 19:47:19.248002 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.247985    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"
Jul 28 19:47:19.255281 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.255254    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"
Jul 28 19:47:19.260438 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.260420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:19.263219 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.263184    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:19.263998 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.263980    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"
Jul 28 19:47:19.364207 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.364154    2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log" limit=1024
Jul 28 19:47:19.377303 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.377201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-j66nk"
Jul 28 19:47:19.422789 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.421076    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf"]
Jul 28 19:47:19.424088 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.424060    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"
Jul 28 19:47:19.425720 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:19.425689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d48cb85_c537_4a11_90f7_1da5815cc0ba.slice/crio-956cd902c246125d9cb1160f52a7c39c2bbfefbd84c44007c76de383bc35d370 WatchSource:0}: Error finding container 956cd902c246125d9cb1160f52a7c39c2bbfefbd84c44007c76de383bc35d370: Status 404 returned error can't find the container with id 956cd902c246125d9cb1160f52a7c39c2bbfefbd84c44007c76de383bc35d370
Jul 28 19:47:19.451037 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.450952    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp"]
Jul 28 19:47:19.456570 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:19.456532    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e30e17e_6f4e_4d92_a1f9_19bdb5b6d0bd.slice/crio-a8db9e4933e9add7b1cfabf32373853a37199f017e7b8511c8e69328f8d31e4b WatchSource:0}: Error finding container a8db9e4933e9add7b1cfabf32373853a37199f017e7b8511c8e69328f8d31e4b: Status 404 returned error can't find the container with id a8db9e4933e9add7b1cfabf32373853a37199f017e7b8511c8e69328f8d31e4b
Jul 28 19:47:19.466794 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.466757    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2"]
Jul 28 19:47:19.473905 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:19.471211    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0377c7ec_588d_42a2_aed1_e21b9f626411.slice/crio-145ae7e72a0aec6b067123e43e9f5727c969d24881d7bfc8643e95f24ed08376 WatchSource:0}: Error finding container 145ae7e72a0aec6b067123e43e9f5727c969d24881d7bfc8643e95f24ed08376: Status 404 returned error can't find the container with id 145ae7e72a0aec6b067123e43e9f5727c969d24881d7bfc8643e95f24ed08376
Jul 28 19:47:19.491769 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.491338    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-nnfhr"]
Jul 28 19:47:19.492906 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:19.492869    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda270ba75_d93d_4d05_ab56_523a14f234a9.slice/crio-8953cc1692bcc986426f6b4e28c4153d3591a08a0b2ef958e38b636d29bc04c8 WatchSource:0}: Error finding container 8953cc1692bcc986426f6b4e28c4153d3591a08a0b2ef958e38b636d29bc04c8: Status 404 returned error can't find the container with id 8953cc1692bcc986426f6b4e28c4153d3591a08a0b2ef958e38b636d29bc04c8
Jul 28 19:47:19.546247 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.546202    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-j66nk"]
Jul 28 19:47:19.549446 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:19.549421    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adb22c5_5525_40b6_a40c_42a13c6abd3a.slice/crio-0cf34641b091f466e60e2fd6acc8751a5dacdde70a8c92efe747b3633792c5c5 WatchSource:0}: Error finding container 0cf34641b091f466e60e2fd6acc8751a5dacdde70a8c92efe747b3633792c5c5: Status 404 returned error can't find the container with id 0cf34641b091f466e60e2fd6acc8751a5dacdde70a8c92efe747b3633792c5c5
Jul 28 19:47:19.591334 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.591155    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd"]
Jul 28 19:47:19.593619 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:19.593593    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98e6895_f9a4_43bd_8cdd_da4b44cc5c95.slice/crio-0c0166991960b876881c1023ac8c84d6c53ce3bee8de09fb0d9782048021f132 WatchSource:0}: Error finding container 0c0166991960b876881c1023ac8c84d6c53ce3bee8de09fb0d9782048021f132: Status 404 returned error can't find the container with id 0c0166991960b876881c1023ac8c84d6c53ce3bee8de09fb0d9782048021f132
Jul 28 19:47:19.627308 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.627279    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:19.627440 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.627314    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.627440 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.627346    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:19.627440 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.627408    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:47:19.627440 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.627429    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 19:47:19.627598 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.627460    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.627438931 +0000 UTC m=+43.527655755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : configmap references non-existent config key: service-ca.crt
Jul 28 19:47:19.627598 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.627488    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert podName:ad0030e8-c33f-49c3-bb42-d0b809f19f98 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.627476477 +0000 UTC m=+43.527693300 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-gsjpt" (UID: "ad0030e8-c33f-49c3-bb42-d0b809f19f98") : secret "networking-console-plugin-cert" not found
Jul 28 19:47:19.627598 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.627504    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.627494326 +0000 UTC m=+43.527711151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : secret "router-metrics-certs-default" not found
Jul 28 19:47:19.728199 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.728109    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:19.728199 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.728195    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:19.728403 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.728287    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 19:47:19.728403 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.728298    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:19.728403 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.728358    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls podName:9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.728340953 +0000 UTC m=+43.628557772 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wq9qq" (UID: "9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6") : secret "samples-operator-tls" not found
Jul 28 19:47:19.728403 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:19.728373    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls podName:f985925d-7690-4f3d-b33d-d12804131a78 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:20.728366785 +0000 UTC m=+43.628583651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j8m4d" (UID: "f985925d-7690-4f3d-b33d-d12804131a78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:19.943255 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.943217    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd" event={"ID":"a98e6895-f9a4-43bd-8cdd-da4b44cc5c95","Type":"ContainerStarted","Data":"0c0166991960b876881c1023ac8c84d6c53ce3bee8de09fb0d9782048021f132"}
Jul 28 19:47:19.944304 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.944282    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j66nk" event={"ID":"3adb22c5-5525-40b6-a40c-42a13c6abd3a","Type":"ContainerStarted","Data":"0cf34641b091f466e60e2fd6acc8751a5dacdde70a8c92efe747b3633792c5c5"}
Jul 28 19:47:19.945217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.945195    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2" event={"ID":"0377c7ec-588d-42a2-aed1-e21b9f626411","Type":"ContainerStarted","Data":"145ae7e72a0aec6b067123e43e9f5727c969d24881d7bfc8643e95f24ed08376"}
Jul 28 19:47:19.946144 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.946121    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp" event={"ID":"3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd","Type":"ContainerStarted","Data":"a8db9e4933e9add7b1cfabf32373853a37199f017e7b8511c8e69328f8d31e4b"}
Jul 28 19:47:19.947140 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.947114    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf" event={"ID":"9d48cb85-c537-4a11-90f7-1da5815cc0ba","Type":"ContainerStarted","Data":"956cd902c246125d9cb1160f52a7c39c2bbfefbd84c44007c76de383bc35d370"}
Jul 28 19:47:19.948082 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:19.948063    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" event={"ID":"a270ba75-d93d-4d05-ab56-523a14f234a9","Type":"ContainerStarted","Data":"8953cc1692bcc986426f6b4e28c4153d3591a08a0b2ef958e38b636d29bc04c8"}
Jul 28 19:47:20.636225 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.636183    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:20.636433 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.636242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:20.636433 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.636296    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:20.636568 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.636545    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 19:47:20.636635 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.636624    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:22.636602923 +0000 UTC m=+45.536819757 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : secret "router-metrics-certs-default" not found
Jul 28 19:47:20.636716 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.636703    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:47:20.636767 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.636747    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert podName:ad0030e8-c33f-49c3-bb42-d0b809f19f98 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:22.636734829 +0000 UTC m=+45.536951663 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-gsjpt" (UID: "ad0030e8-c33f-49c3-bb42-d0b809f19f98") : secret "networking-console-plugin-cert" not found
Jul 28 19:47:20.636836 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.636823    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:22.6368128 +0000 UTC m=+45.537029624 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : configmap references non-existent config key: service-ca.crt
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.737439    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.737558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.737594    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:20.737640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.737845    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.737915    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls podName:9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:22.737895778 +0000 UTC m=+45.638112603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wq9qq" (UID: "9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6") : secret "samples-operator-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738337    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738390    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls podName:f985925d-7690-4f3d-b33d-d12804131a78 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:22.738374937 +0000 UTC m=+45.638591762 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j8m4d" (UID: "f985925d-7690-4f3d-b33d-d12804131a78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738457    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738492    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls podName:fd16eed2-e3aa-4672-bdfc-c9c10b376e9b nodeName:}" failed. No retries permitted until 2026-07-28 19:47:28.738480093 +0000 UTC m=+51.638696928 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls") pod "dns-default-jh7xz" (UID: "fd16eed2-e3aa-4672-bdfc-c9c10b376e9b") : secret "dns-default-metrics-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738554    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738567    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf84dbb-dcxv5: secret "image-registry-tls" not found
Jul 28 19:47:20.738636 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:20.738600    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls podName:34065413-44cf-4622-9f6f-be947f1d7c82 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:28.738589236 +0000 UTC m=+51.638806071 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls") pod "image-registry-698cf84dbb-dcxv5" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82") : secret "image-registry-tls" not found
Jul 28 19:47:22.657815 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:22.657768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:22.657841    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:22.657873    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.657923    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.658004    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert podName:ad0030e8-c33f-49c3-bb42-d0b809f19f98 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:26.657983503 +0000 UTC m=+49.558200327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-gsjpt" (UID: "ad0030e8-c33f-49c3-bb42-d0b809f19f98") : secret "networking-console-plugin-cert" not found
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.658024    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:26.65801423 +0000 UTC m=+49.558231052 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : configmap references non-existent config key: service-ca.crt
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.658076    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 19:47:22.658298 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.658139    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:26.658122652 +0000 UTC m=+49.558339485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : secret "router-metrics-certs-default" not found
Jul 28 19:47:22.758629 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:22.758587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:22.758796 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:22.758736    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:22.758796 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.758754    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 19:47:22.758885 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.758824    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls podName:9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:26.758804022 +0000 UTC m=+49.659020868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wq9qq" (UID: "9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6") : secret "samples-operator-tls" not found
Jul 28 19:47:22.758885 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.758856    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:22.758957 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:22.758890    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls podName:f985925d-7690-4f3d-b33d-d12804131a78 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:26.758878678 +0000 UTC m=+49.659095498 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j8m4d" (UID: "f985925d-7690-4f3d-b33d-d12804131a78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:26.696341 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.696299    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.696464    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.696476    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.696494    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.696544    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:34.696525102 +0000 UTC m=+57.596741949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : secret "router-metrics-certs-default" not found
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.696604    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.696620    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:34.696603083 +0000 UTC m=+57.596819919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : configmap references non-existent config key: service-ca.crt
Jul 28 19:47:26.696842 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.696647    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert podName:ad0030e8-c33f-49c3-bb42-d0b809f19f98 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:34.696636786 +0000 UTC m=+57.596853606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-gsjpt" (UID: "ad0030e8-c33f-49c3-bb42-d0b809f19f98") : secret "networking-console-plugin-cert" not found
Jul 28 19:47:26.797998 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.797951    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:26.798193 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.798104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:26.798193 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.798135    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:26.798318 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.798215    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls podName:f985925d-7690-4f3d-b33d-d12804131a78 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:34.798194711 +0000 UTC m=+57.698411544 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j8m4d" (UID: "f985925d-7690-4f3d-b33d-d12804131a78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:26.798318 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.798225    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 19:47:26.798318 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:26.798292    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls podName:9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:34.798260632 +0000 UTC m=+57.698477455 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wq9qq" (UID: "9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6") : secret "samples-operator-tls" not found
Jul 28 19:47:26.971119 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.971033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd" event={"ID":"a98e6895-f9a4-43bd-8cdd-da4b44cc5c95","Type":"ContainerStarted","Data":"d2d423766b583e76dc26e43971b74ab1409f16162e5a79c63770d416277e25ea"}
Jul 28 19:47:26.972875 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.972847    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j66nk" event={"ID":"3adb22c5-5525-40b6-a40c-42a13c6abd3a","Type":"ContainerStarted","Data":"45b6fc466cca940d7fac7bef75a39d093f5dff4b1c0a129ecf1cbb15c085a6b5"}
Jul 28 19:47:26.974910 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.974688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2" event={"ID":"0377c7ec-588d-42a2-aed1-e21b9f626411","Type":"ContainerStarted","Data":"f9da7ffa2dbd44b034ebaedd615ea09f9a0029eb8cda6802baf0c101cc1965e1"}
Jul 28 19:47:26.976571 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.976548    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp" event={"ID":"3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd","Type":"ContainerStarted","Data":"b7c04de02d488cbf0ecff7a0e18a188d384fbdfa57ab77219c13509fc6055a0a"}
Jul 28 19:47:26.978590 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.978560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf" event={"ID":"9d48cb85-c537-4a11-90f7-1da5815cc0ba","Type":"ContainerStarted","Data":"f3a90f9318253b98ce5f0e0ab3b9ae9686d69c7ef135107ceca3feaf16974038"}
Jul 28 19:47:26.980577 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.980555    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/0.log"
Jul 28 19:47:26.980674 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.980600    2578 generic.go:356] "Generic (PLEG): container finished" podID="a270ba75-d93d-4d05-ab56-523a14f234a9" containerID="60ee245f5899fa69935c3a8d0ef49f71d2aba6e12382ae76f70f756b30231c59" exitCode=255
Jul 28 19:47:26.980674 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.980648    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" event={"ID":"a270ba75-d93d-4d05-ab56-523a14f234a9","Type":"ContainerDied","Data":"60ee245f5899fa69935c3a8d0ef49f71d2aba6e12382ae76f70f756b30231c59"}
Jul 28 19:47:26.980848 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:26.980832    2578 scope.go:117] "RemoveContainer" containerID="60ee245f5899fa69935c3a8d0ef49f71d2aba6e12382ae76f70f756b30231c59"
Jul 28 19:47:27.002058 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.001998    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-jq4rd" podStartSLOduration=1.664044681 podStartE2EDuration="8.001979729s" totalImagesPullingTime="6.337935048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:26.990141354 +0000 UTC m=+49.890358197" watchObservedRunningTime="2026-07-28 19:47:27.001979729 +0000 UTC m=+49.902196572"
Jul 28 19:47:27.002870 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.002841    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l9sw5"]
Jul 28 19:47:27.010884 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.010827    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-cfccp" podStartSLOduration=6.873254183 podStartE2EDuration="9.010808773s" totalImagesPullingTime="2.13755459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:27.010042565 +0000 UTC m=+49.910259445" watchObservedRunningTime="2026-07-28 19:47:27.010808773 +0000 UTC m=+49.911025616"
Jul 28 19:47:27.027430 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.025717    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.034348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.031507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9sw5"]
Jul 28 19:47:27.034348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.031728    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t6cqg\""
Jul 28 19:47:27.034348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.031756    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 19:47:27.034348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.032119    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 19:47:27.034348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.032785    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-j66nk" podStartSLOduration=2.6452479330000003 podStartE2EDuration="9.032767771s" totalImagesPullingTime="6.387519838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:27.032301009 +0000 UTC m=+49.932517852" watchObservedRunningTime="2026-07-28 19:47:27.032767771 +0000 UTC m=+49.932984614"
Jul 28 19:47:27.052242 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.052195    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf" podStartSLOduration=2.547557859 podStartE2EDuration="9.052178132s" totalImagesPullingTime="6.504620273s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:27.051013607 +0000 UTC m=+49.951230450" watchObservedRunningTime="2026-07-28 19:47:27.052178132 +0000 UTC m=+49.952394965"
Jul 28 19:47:27.088426 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.088366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2" podStartSLOduration=2.630215427 podStartE2EDuration="9.088344381s" totalImagesPullingTime="6.458128954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:27.086631505 +0000 UTC m=+49.986848347" watchObservedRunningTime="2026-07-28 19:47:27.088344381 +0000 UTC m=+49.988561224"
Jul 28 19:47:27.104486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.101986    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.104486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.102125    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-crio-socket\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.104486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.102206    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.104486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.102322    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4ff\" (UniqueName: \"kubernetes.io/projected/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-api-access-6z4ff\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.104486 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.102359    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-data-volume\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203232 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203196    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203434 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-crio-socket\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203434 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.203373    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:47:27.203541 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.203445    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:27.703423065 +0000 UTC m=+50.603639901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : configmap "kube-rbac-proxy" not found
Jul 28 19:47:27.203541 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-crio-socket\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203541 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203676 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4ff\" (UniqueName: \"kubernetes.io/projected/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-api-access-6z4ff\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203676 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203588    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-data-volume\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.203676 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.203631    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:27.203778 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.203684    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:27.703667471 +0000 UTC m=+50.603884305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:27.203985 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.203969    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-data-volume\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.214517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.214483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4ff\" (UniqueName: \"kubernetes.io/projected/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-api-access-6z4ff\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.708556 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.708519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.708988 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.708627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:27.708988 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.708660    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:47:27.708988 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.708745    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:28.708723151 +0000 UTC m=+51.608939975 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : configmap "kube-rbac-proxy" not found
Jul 28 19:47:27.708988 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.708776    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:27.708988 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.708816    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:28.708804856 +0000 UTC m=+51.609021677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:27.985612 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.985536    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:47:27.985965 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.985949    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/0.log"
Jul 28 19:47:27.986026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.985983    2578 generic.go:356] "Generic (PLEG): container finished" podID="a270ba75-d93d-4d05-ab56-523a14f234a9" containerID="56d98ac9da9f416f507383f9bc709162401343fcd91f40e785bd51eab0c020dd" exitCode=255
Jul 28 19:47:27.986084 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.986016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" event={"ID":"a270ba75-d93d-4d05-ab56-523a14f234a9","Type":"ContainerDied","Data":"56d98ac9da9f416f507383f9bc709162401343fcd91f40e785bd51eab0c020dd"}
Jul 28 19:47:27.986136 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.986091    2578 scope.go:117] "RemoveContainer" containerID="60ee245f5899fa69935c3a8d0ef49f71d2aba6e12382ae76f70f756b30231c59"
Jul 28 19:47:27.986371 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:27.986339    2578 scope.go:117] "RemoveContainer" containerID="56d98ac9da9f416f507383f9bc709162401343fcd91f40e785bd51eab0c020dd"
Jul 28 19:47:27.986564 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:27.986538    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-nnfhr_openshift-console-operator(a270ba75-d93d-4d05-ab56-523a14f234a9)\"" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" podUID="a270ba75-d93d-4d05-ab56-523a14f234a9"
Jul 28 19:47:28.719690 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.719651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:28.720078 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.719754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:28.720078 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.719782    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 19:47:28.720078 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.719852    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:30.719834401 +0000 UTC m=+53.620051238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : configmap "kube-rbac-proxy" not found
Jul 28 19:47:28.720078 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.719851    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:28.720078 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.719889    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:30.719880077 +0000 UTC m=+53.620096905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:28.820762 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.820719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:28.820762 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.820768    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:28.820964 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.820866    2578 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 19:47:28.820964 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.820877    2578 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-698cf84dbb-dcxv5: secret "image-registry-tls" not found
Jul 28 19:47:28.820964 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.820887    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 19:47:28.820964 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.820927    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls podName:34065413-44cf-4622-9f6f-be947f1d7c82 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:44.820914243 +0000 UTC m=+67.721131063 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls") pod "image-registry-698cf84dbb-dcxv5" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82") : secret "image-registry-tls" not found
Jul 28 19:47:28.820964 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.820946    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls podName:fd16eed2-e3aa-4672-bdfc-c9c10b376e9b nodeName:}" failed. No retries permitted until 2026-07-28 19:47:44.820932931 +0000 UTC m=+67.721149750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls") pod "dns-default-jh7xz" (UID: "fd16eed2-e3aa-4672-bdfc-c9c10b376e9b") : secret "dns-default-metrics-tls" not found
Jul 28 19:47:28.828336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.828316    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx69x_e77e8b7e-663f-417e-8e2c-73683e752699/dns-node-resolver/0.log"
Jul 28 19:47:28.990111 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.990032    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:47:28.990387 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:28.990371    2578 scope.go:117] "RemoveContainer" containerID="56d98ac9da9f416f507383f9bc709162401343fcd91f40e785bd51eab0c020dd"
Jul 28 19:47:28.990553 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:28.990530    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-nnfhr_openshift-console-operator(a270ba75-d93d-4d05-ab56-523a14f234a9)\"" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" podUID="a270ba75-d93d-4d05-ab56-523a14f234a9"
Jul 28 19:47:29.261229 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:29.261193    2578 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:29.261395 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:29.261306    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:29.427752 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:29.427728    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-974jf_986541f9-ad8a-4755-b6bc-37e010abdbdb/node-ca/0.log"
Jul 28 19:47:29.993745 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:29.993718    2578 scope.go:117] "RemoveContainer" containerID="56d98ac9da9f416f507383f9bc709162401343fcd91f40e785bd51eab0c020dd"
Jul 28 19:47:29.994124 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:29.993873    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-nnfhr_openshift-console-operator(a270ba75-d93d-4d05-ab56-523a14f234a9)\"" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" podUID="a270ba75-d93d-4d05-ab56-523a14f234a9"
Jul 28 19:47:30.228717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:30.228681    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6v2wp_33895c8b-396f-428a-98ae-edb4ce1cf0c4/serve-healthcheck-canary/0.log"
Jul 28 19:47:30.739174 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:30.739126    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:30.739444 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:30.739218    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:30.739444 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:30.739364    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:30.739444 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:30.739434    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:34.739418348 +0000 UTC m=+57.639635168 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:30.739676 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:30.739658    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:34.775894 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:34.775852    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:34.775894 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:34.775901    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.775999    2578 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.776017    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:50.77599886 +0000 UTC m=+73.676215683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : configmap references non-existent config key: service-ca.crt
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:34.776034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.776064    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert podName:ad0030e8-c33f-49c3-bb42-d0b809f19f98 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:50.776051194 +0000 UTC m=+73.676268017 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-gsjpt" (UID: "ad0030e8-c33f-49c3-bb42-d0b809f19f98") : secret "networking-console-plugin-cert" not found
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.776104    2578 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:34.776113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.776139    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs podName:6cb891fa-f2eb-429d-abcd-ee896160e98a nodeName:}" failed. No retries permitted until 2026-07-28 19:47:50.776127042 +0000 UTC m=+73.676343861 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs") pod "router-default-86b574f647-594xn" (UID: "6cb891fa-f2eb-429d-abcd-ee896160e98a") : secret "router-metrics-certs-default" not found
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.776176    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:34.776401 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.776205    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls podName:fec949f3-4c83-40dd-8b20-73eca8c3d8ef nodeName:}" failed. No retries permitted until 2026-07-28 19:47:42.776197828 +0000 UTC m=+65.676414667 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9sw5" (UID: "fec949f3-4c83-40dd-8b20-73eca8c3d8ef") : secret "insights-runtime-extractor-tls" not found
Jul 28 19:47:34.877216 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:34.877173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:34.877441 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:34.877293    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:34.877441 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.877362    2578 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:34.877441 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.877408    2578 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 19:47:34.877603 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.877454    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls podName:f985925d-7690-4f3d-b33d-d12804131a78 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:50.877433629 +0000 UTC m=+73.777650466 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-j8m4d" (UID: "f985925d-7690-4f3d-b33d-d12804131a78") : secret "cluster-monitoring-operator-tls" not found
Jul 28 19:47:34.877603 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:47:34.877474    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls podName:9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6 nodeName:}" failed. No retries permitted until 2026-07-28 19:47:50.877465292 +0000 UTC m=+73.777682114 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls") pod "cluster-samples-operator-6748489656-wq9qq" (UID: "9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6") : secret "samples-operator-tls" not found
Jul 28 19:47:37.895560 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:37.895528    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cbv5w"
Jul 28 19:47:41.645770 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:41.645741    2578 scope.go:117] "RemoveContainer" containerID="56d98ac9da9f416f507383f9bc709162401343fcd91f40e785bd51eab0c020dd"
Jul 28 19:47:42.032053 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.032029    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:47:42.032204 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.032101    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" event={"ID":"a270ba75-d93d-4d05-ab56-523a14f234a9","Type":"ContainerStarted","Data":"103f76e64199358d8752bff16133ad8a815a8b7ffb20f08142eb6fddc4358bb4"}
Jul 28 19:47:42.032415 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.032395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:42.037482 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.037456    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr"
Jul 28 19:47:42.048814 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.048759    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-nnfhr" podStartSLOduration=17.609854202 podStartE2EDuration="24.048738061s" totalImagesPullingTime="6.438883859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:42.048470673 +0000 UTC m=+64.948687531" watchObservedRunningTime="2026-07-28 19:47:42.048738061 +0000 UTC m=+64.948954905"
Jul 28 19:47:42.851284 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.851235    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:42.853695 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.853676    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/fec949f3-4c83-40dd-8b20-73eca8c3d8ef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9sw5\" (UID: \"fec949f3-4c83-40dd-8b20-73eca8c3d8ef\") " pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:42.952745 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.952717    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-t6cqg\""
Jul 28 19:47:42.960648 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:42.960627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9sw5"
Jul 28 19:47:43.086015 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.085984    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9sw5"]
Jul 28 19:47:43.089669 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:43.089640    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec949f3_4c83_40dd_8b20_73eca8c3d8ef.slice/crio-9bf9a3bac945892b0a88975068bfda378b28f2b5f47619ef2d1262f216bd50ba WatchSource:0}: Error finding container 9bf9a3bac945892b0a88975068bfda378b28f2b5f47619ef2d1262f216bd50ba: Status 404 returned error can't find the container with id 9bf9a3bac945892b0a88975068bfda378b28f2b5f47619ef2d1262f216bd50ba
Jul 28 19:47:43.253953 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.253854    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:43.256708 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.256685    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 19:47:43.267232 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.267205    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03c8c9fd-d068-468d-a118-e2a49e136cee-metrics-certs\") pod \"network-metrics-daemon-m9tq9\" (UID: \"03c8c9fd-d068-468d-a118-e2a49e136cee\") " pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:43.354727 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.354690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:43.354906 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.354733    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:43.357253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.357228    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 19:47:43.357382 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.357343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps65r\" (UniqueName: \"kubernetes.io/projected/86914da0-7e38-465d-bf4b-1207416887a2-kube-api-access-ps65r\") pod \"network-check-target-n995h\" (UID: \"86914da0-7e38-465d-bf4b-1207416887a2\") " pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:43.367302 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.367279    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4cad5271-713f-4162-b9ad-00cdc9dd0221-original-pull-secret\") pod \"global-pull-secret-syncer-lt7fb\" (UID: \"4cad5271-713f-4162-b9ad-00cdc9dd0221\") " pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:43.460106 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.460072    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lhtrs\""
Jul 28 19:47:43.463011 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.462989    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lt7fb"
Jul 28 19:47:43.467975 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.467956    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:43.470975 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.470956    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-l4wgt\""
Jul 28 19:47:43.478889 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.478866    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m9tq9"
Jul 28 19:47:43.616512 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.616429    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lt7fb"]
Jul 28 19:47:43.621256 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:43.621187    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cad5271_713f_4162_b9ad_00cdc9dd0221.slice/crio-4b8ce3221b50260baaad8b7a81ab54c06f5f184444609c37df2efb56fd4b09e6 WatchSource:0}: Error finding container 4b8ce3221b50260baaad8b7a81ab54c06f5f184444609c37df2efb56fd4b09e6: Status 404 returned error can't find the container with id 4b8ce3221b50260baaad8b7a81ab54c06f5f184444609c37df2efb56fd4b09e6
Jul 28 19:47:43.639526 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.638605    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-n995h"]
Jul 28 19:47:43.642956 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:43.642921    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86914da0_7e38_465d_bf4b_1207416887a2.slice/crio-df431b8d963396a58bf0e05479ecc9a20156e6661a6b85833789d982ba4a7211 WatchSource:0}: Error finding container df431b8d963396a58bf0e05479ecc9a20156e6661a6b85833789d982ba4a7211: Status 404 returned error can't find the container with id df431b8d963396a58bf0e05479ecc9a20156e6661a6b85833789d982ba4a7211
Jul 28 19:47:43.665056 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:43.665030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m9tq9"]
Jul 28 19:47:43.668628 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:43.668602    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c8c9fd_d068_468d_a118_e2a49e136cee.slice/crio-209079431c0c5fd3f886edd965109c361eefeeef8dceffa7e19b17f5efd2ebd0 WatchSource:0}: Error finding container 209079431c0c5fd3f886edd965109c361eefeeef8dceffa7e19b17f5efd2ebd0: Status 404 returned error can't find the container with id 209079431c0c5fd3f886edd965109c361eefeeef8dceffa7e19b17f5efd2ebd0
Jul 28 19:47:44.042447 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.042333    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m9tq9" event={"ID":"03c8c9fd-d068-468d-a118-e2a49e136cee","Type":"ContainerStarted","Data":"209079431c0c5fd3f886edd965109c361eefeeef8dceffa7e19b17f5efd2ebd0"}
Jul 28 19:47:44.044652 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.044615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n995h" event={"ID":"86914da0-7e38-465d-bf4b-1207416887a2","Type":"ContainerStarted","Data":"13929498b4b65f9650e1e6fdb51ca21eec13dd13b66085f6154e6b2a1078a4c3"}
Jul 28 19:47:44.044772 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.044671    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-n995h" event={"ID":"86914da0-7e38-465d-bf4b-1207416887a2","Type":"ContainerStarted","Data":"df431b8d963396a58bf0e05479ecc9a20156e6661a6b85833789d982ba4a7211"}
Jul 28 19:47:44.044772 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.044724    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:47:44.046069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.046040    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lt7fb" event={"ID":"4cad5271-713f-4162-b9ad-00cdc9dd0221","Type":"ContainerStarted","Data":"4b8ce3221b50260baaad8b7a81ab54c06f5f184444609c37df2efb56fd4b09e6"}
Jul 28 19:47:44.047895 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.047870    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sw5" event={"ID":"fec949f3-4c83-40dd-8b20-73eca8c3d8ef","Type":"ContainerStarted","Data":"9bf9a3bac945892b0a88975068bfda378b28f2b5f47619ef2d1262f216bd50ba"}
Jul 28 19:47:44.873899 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.873834    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:44.874084 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.874004    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:44.876726 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.876677    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fd16eed2-e3aa-4672-bdfc-c9c10b376e9b-metrics-tls\") pod \"dns-default-jh7xz\" (UID: \"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b\") " pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:44.879338 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:44.879292    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"image-registry-698cf84dbb-dcxv5\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") " pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:45.042404 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.042375    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c54t5\""
Jul 28 19:47:45.050294 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.050257    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:45.051950 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.051930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wfpzj\""
Jul 28 19:47:45.055304 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.055125    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m9tq9" event={"ID":"03c8c9fd-d068-468d-a118-e2a49e136cee","Type":"ContainerStarted","Data":"9ea4d9b02d3bcbe765e032c6f40d322c65edbff6cc0f0c9011a4793535e1a38c"}
Jul 28 19:47:45.059986 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.059951    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:45.240310 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.240215    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-n995h" podStartSLOduration=67.240190691 podStartE2EDuration="1m7.240190691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:44.063657466 +0000 UTC m=+66.963874323" watchObservedRunningTime="2026-07-28 19:47:45.240190691 +0000 UTC m=+68.140407534"
Jul 28 19:47:45.240978 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.240953    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-698cf84dbb-dcxv5"]
Jul 28 19:47:45.245602 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:45.245571    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34065413_44cf_4622_9f6f_be947f1d7c82.slice/crio-dc9e583fd92130a6505675018a91a229a64187a625bb1c6b0c97c13d8c7d4bc9 WatchSource:0}: Error finding container dc9e583fd92130a6505675018a91a229a64187a625bb1c6b0c97c13d8c7d4bc9: Status 404 returned error can't find the container with id dc9e583fd92130a6505675018a91a229a64187a625bb1c6b0c97c13d8c7d4bc9
Jul 28 19:47:45.261833 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:45.261806    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jh7xz"]
Jul 28 19:47:45.264878 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:45.264850    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd16eed2_e3aa_4672_bdfc_c9c10b376e9b.slice/crio-58b7136fa3a54863fd374ab0d87b71435f6b583c94ff5b06055787427e0575d7 WatchSource:0}: Error finding container 58b7136fa3a54863fd374ab0d87b71435f6b583c94ff5b06055787427e0575d7: Status 404 returned error can't find the container with id 58b7136fa3a54863fd374ab0d87b71435f6b583c94ff5b06055787427e0575d7
Jul 28 19:47:46.063642 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.063100    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m9tq9" event={"ID":"03c8c9fd-d068-468d-a118-e2a49e136cee","Type":"ContainerStarted","Data":"1d4fed06bbb5265871e2c66a97e217f3b9510f61df4b1a9569dc32ac3deea0b2"}
Jul 28 19:47:46.067585 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.066918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5" event={"ID":"34065413-44cf-4622-9f6f-be947f1d7c82","Type":"ContainerStarted","Data":"dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed"}
Jul 28 19:47:46.067585 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.066972    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5" event={"ID":"34065413-44cf-4622-9f6f-be947f1d7c82","Type":"ContainerStarted","Data":"dc9e583fd92130a6505675018a91a229a64187a625bb1c6b0c97c13d8c7d4bc9"}
Jul 28 19:47:46.067585 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.067317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:47:46.068749 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.068545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jh7xz" event={"ID":"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b","Type":"ContainerStarted","Data":"58b7136fa3a54863fd374ab0d87b71435f6b583c94ff5b06055787427e0575d7"}
Jul 28 19:47:46.078782 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.078723    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m9tq9" podStartSLOduration=66.851710453 podStartE2EDuration="1m8.078703705s" totalImagesPullingTime="1.226993252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:46.077869835 +0000 UTC m=+68.978086694" watchObservedRunningTime="2026-07-28 19:47:46.078703705 +0000 UTC m=+68.978920548"
Jul 28 19:47:46.098327 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:46.098238    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5" podStartSLOduration=67.098220603 podStartE2EDuration="1m7.098220603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:46.098094724 +0000 UTC m=+68.998311569" watchObservedRunningTime="2026-07-28 19:47:46.098220603 +0000 UTC m=+68.998437446"
Jul 28 19:47:50.832788 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.832757    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:50.833182 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.832808    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:50.833182 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.832844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:50.833800 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.833774    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cb891fa-f2eb-429d-abcd-ee896160e98a-service-ca-bundle\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:50.835626 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.835581    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cb891fa-f2eb-429d-abcd-ee896160e98a-metrics-certs\") pod \"router-default-86b574f647-594xn\" (UID: \"6cb891fa-f2eb-429d-abcd-ee896160e98a\") " pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:50.836560 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.836510    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ad0030e8-c33f-49c3-bb42-d0b809f19f98-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-gsjpt\" (UID: \"ad0030e8-c33f-49c3-bb42-d0b809f19f98\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:50.934323 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.933746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:50.934323 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.933844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:50.938165 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.937944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f985925d-7690-4f3d-b33d-d12804131a78-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-j8m4d\" (UID: \"f985925d-7690-4f3d-b33d-d12804131a78\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:50.938365 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:50.938342    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-wq9qq\" (UID: \"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:51.074288 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.074179    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-ssps7\""
Jul 28 19:47:51.082151 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.082125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:51.088560 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.088094    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lt7fb" event={"ID":"4cad5271-713f-4162-b9ad-00cdc9dd0221","Type":"ContainerStarted","Data":"b4af29e9a5696b8eb5c042fa776bf28e39165870b1bab9f85ed3f2a9a0758744"}
Jul 28 19:47:51.089744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.089718    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sw5" event={"ID":"fec949f3-4c83-40dd-8b20-73eca8c3d8ef","Type":"ContainerStarted","Data":"8f7ca6d8e9e198514d72f24122e2cc293e70fa46891a127e1e32abfd955750e7"}
Jul 28 19:47:51.091679 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.091653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jh7xz" event={"ID":"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b","Type":"ContainerStarted","Data":"0af1ce24a4ee278d04b76e63d8fef1444d25b14e251df9b9479197f41cd80a25"}
Jul 28 19:47:51.091804 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.091704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jh7xz" event={"ID":"fd16eed2-e3aa-4672-bdfc-c9c10b376e9b","Type":"ContainerStarted","Data":"e526dbde9bc3a766a6fd3ad429855cb7ad7c594848ba31af8d23cf2f4a432d93"}
Jul 28 19:47:51.091804 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.091795    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:47:51.103937 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.103884    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lt7fb" podStartSLOduration=64.898444128 podStartE2EDuration="1m12.103863901s" totalImagesPullingTime="7.205419773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:46:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:51.102622588 +0000 UTC m=+74.002839431" watchObservedRunningTime="2026-07-28 19:47:51.103863901 +0000 UTC m=+74.004080742"
Jul 28 19:47:51.118034 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.117963    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jh7xz" podStartSLOduration=33.579195845 podStartE2EDuration="39.117947145s" totalImagesPullingTime="5.5387513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:51.117692267 +0000 UTC m=+74.017909109" watchObservedRunningTime="2026-07-28 19:47:51.117947145 +0000 UTC m=+74.018163988"
Jul 28 19:47:51.121383 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.121359    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nfftp\""
Jul 28 19:47:51.129973 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.129935    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"
Jul 28 19:47:51.144966 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.144936    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-5lnhb\""
Jul 28 19:47:51.152450 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.152402    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"
Jul 28 19:47:51.154856 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.154695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-52l28\""
Jul 28 19:47:51.163179 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.163159    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"
Jul 28 19:47:51.240179 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.239425    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-86b574f647-594xn"]
Jul 28 19:47:51.294127 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.294035    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt"]
Jul 28 19:47:51.327166 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.326342    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq"]
Jul 28 19:47:51.349877 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:51.345941    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d"]
Jul 28 19:47:51.349877 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:51.348095    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf985925d_7690_4f3d_b33d_d12804131a78.slice/crio-a316274d0577797f3eb0cea94034dc9c757d861c2cde38f4015a07f8f7cdc0d4 WatchSource:0}: Error finding container a316274d0577797f3eb0cea94034dc9c757d861c2cde38f4015a07f8f7cdc0d4: Status 404 returned error can't find the container with id a316274d0577797f3eb0cea94034dc9c757d861c2cde38f4015a07f8f7cdc0d4
Jul 28 19:47:52.105771 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.105665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d" event={"ID":"f985925d-7690-4f3d-b33d-d12804131a78","Type":"ContainerStarted","Data":"a316274d0577797f3eb0cea94034dc9c757d861c2cde38f4015a07f8f7cdc0d4"}
Jul 28 19:47:52.108569 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.108050    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86b574f647-594xn" event={"ID":"6cb891fa-f2eb-429d-abcd-ee896160e98a","Type":"ContainerStarted","Data":"ab97e991c4d0bc414078eb54b552c29e75bed63cbb15e3fefe4e37568af406db"}
Jul 28 19:47:52.108569 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.108089    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-86b574f647-594xn" event={"ID":"6cb891fa-f2eb-429d-abcd-ee896160e98a","Type":"ContainerStarted","Data":"0d50a812783160fb0eb5f8e2dd4a339d0f7ac901fe61e9eeed20db4c14a5b8a5"}
Jul 28 19:47:52.110502 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.110468    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq" event={"ID":"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6","Type":"ContainerStarted","Data":"54077be27b19dc285bff48c46c5cb40d7d555a8ef4b08665eda53a8ee38f7a7f"}
Jul 28 19:47:52.112783 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.112737    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt" event={"ID":"ad0030e8-c33f-49c3-bb42-d0b809f19f98","Type":"ContainerStarted","Data":"8a0c38f1705f0f1abbda38886d8ab5271c0aba44bdffa6549b32e887159f1064"}
Jul 28 19:47:52.129200 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.129140    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-86b574f647-594xn" podStartSLOduration=34.12911642 podStartE2EDuration="34.12911642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:52.127744497 +0000 UTC m=+75.027961341" watchObservedRunningTime="2026-07-28 19:47:52.12911642 +0000 UTC m=+75.029333305"
Jul 28 19:47:52.787570 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.787535    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"]
Jul 28 19:47:52.810321 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.810288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"]
Jul 28 19:47:52.810490 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.810419    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:52.813734 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.813694    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 19:47:52.814017 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.813994    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 19:47:52.814382 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.814351    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 19:47:52.815012 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.814988    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 28 19:47:52.892931 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.892895    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-kpzrq"]
Jul 28 19:47:52.895525 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.895498    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d98f64cb9-j45cl"]
Jul 28 19:47:52.895915 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.895890    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:47:52.897940 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.897916    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:52.899875 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.899841    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-w68cv\""
Jul 28 19:47:52.899875 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.899869    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 19:47:52.900032 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.899842    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 19:47:52.901305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.900739    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 19:47:52.901305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.901033    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 19:47:52.901305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.901085    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lxsh7\""
Jul 28 19:47:52.901305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.901094    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 19:47:52.901305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.901033    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 19:47:52.901305 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.901210    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 19:47:52.912976 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.912947    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-kpzrq"]
Jul 28 19:47:52.914527 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.914503    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d98f64cb9-j45cl"]
Jul 28 19:47:52.952960 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.952924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsxb\" (UniqueName: \"kubernetes.io/projected/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-kube-api-access-tvsxb\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:52.953146 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.953015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:52.953146 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:52.953044    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-tmp\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.054475 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054384    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vsf\" (UniqueName: \"kubernetes.io/projected/56e2de65-a321-45dd-9ca8-bc13330c4a64-kube-api-access-k9vsf\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.054475 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054445    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdjxn\" (UniqueName: \"kubernetes.io/projected/f18750e0-eeda-45c0-9150-8240779c4755-kube-api-access-fdjxn\") pod \"downloads-7c97dc4555-kpzrq\" (UID: \"f18750e0-eeda-45c0-9150-8240779c4755\") " pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054558    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-tmp\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054581    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-config\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054618    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-serving-cert\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054654    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-oauth-config\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-service-ca\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-oauth-serving-cert\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.054793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.054741    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsxb\" (UniqueName: \"kubernetes.io/projected/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-kube-api-access-tvsxb\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.055256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.055159    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-tmp\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.057708 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.057675    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-klusterlet-config\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.067446 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.067416    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsxb\" (UniqueName: \"kubernetes.io/projected/3f309a2c-89e8-496a-bd16-2a7f1eb853e7-kube-api-access-tvsxb\") pod \"klusterlet-addon-workmgr-58d7d84c75-cwzmm\" (UID: \"3f309a2c-89e8-496a-bd16-2a7f1eb853e7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.082753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.082729    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:53.085739 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.085712    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:53.116872 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.116839    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:53.118226 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.118205    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-86b574f647-594xn"
Jul 28 19:47:53.125811 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.125782    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:47:53.155678 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155649    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-oauth-serving-cert\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.155858 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155724    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vsf\" (UniqueName: \"kubernetes.io/projected/56e2de65-a321-45dd-9ca8-bc13330c4a64-kube-api-access-k9vsf\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.155858 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdjxn\" (UniqueName: \"kubernetes.io/projected/f18750e0-eeda-45c0-9150-8240779c4755-kube-api-access-fdjxn\") pod \"downloads-7c97dc4555-kpzrq\" (UID: \"f18750e0-eeda-45c0-9150-8240779c4755\") " pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:47:53.155858 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155793    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-config\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.155858 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155827    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-serving-cert\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.156066 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155859    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-oauth-config\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.156066 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.155881    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-service-ca\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.156645 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.156600    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-service-ca\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.156856 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.156825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-config\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.157029 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.157008    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-oauth-serving-cert\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.160296 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.160231    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-oauth-config\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.160462 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.160437    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-serving-cert\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.173785 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.173755    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdjxn\" (UniqueName: \"kubernetes.io/projected/f18750e0-eeda-45c0-9150-8240779c4755-kube-api-access-fdjxn\") pod \"downloads-7c97dc4555-kpzrq\" (UID: \"f18750e0-eeda-45c0-9150-8240779c4755\") " pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:47:53.176301 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.176171    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vsf\" (UniqueName: \"kubernetes.io/projected/56e2de65-a321-45dd-9ca8-bc13330c4a64-kube-api-access-k9vsf\") pod \"console-d98f64cb9-j45cl\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") " pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:53.218575 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.218542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:47:53.229111 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:53.229081    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:47:55.672871 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:55.672679    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"]
Jul 28 19:47:56.286281 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:56.286241    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f309a2c_89e8_496a_bd16_2a7f1eb853e7.slice/crio-d89467f4393a57b5ef8204014ab77d6f72aa4db60d1218c39222ab61a449c5fe WatchSource:0}: Error finding container d89467f4393a57b5ef8204014ab77d6f72aa4db60d1218c39222ab61a449c5fe: Status 404 returned error can't find the container with id d89467f4393a57b5ef8204014ab77d6f72aa4db60d1218c39222ab61a449c5fe
Jul 28 19:47:56.406569 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:56.406535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d98f64cb9-j45cl"]
Jul 28 19:47:56.488375 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:56.488341    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e2de65_a321_45dd_9ca8_bc13330c4a64.slice/crio-65a02f69b985c0986c9ea364060ae951a27f9a85a9f2e3d2710ae2d1fea55ec2 WatchSource:0}: Error finding container 65a02f69b985c0986c9ea364060ae951a27f9a85a9f2e3d2710ae2d1fea55ec2: Status 404 returned error can't find the container with id 65a02f69b985c0986c9ea364060ae951a27f9a85a9f2e3d2710ae2d1fea55ec2
Jul 28 19:47:56.676086 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:56.675760    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-kpzrq"]
Jul 28 19:47:56.679756 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:47:56.679719    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf18750e0_eeda_45c0_9150_8240779c4755.slice/crio-3cfb7e065e8880351b93c03e2d661e8f741d2eb2a685f0b90a6ddac59cc5fd0e WatchSource:0}: Error finding container 3cfb7e065e8880351b93c03e2d661e8f741d2eb2a685f0b90a6ddac59cc5fd0e: Status 404 returned error can't find the container with id 3cfb7e065e8880351b93c03e2d661e8f741d2eb2a685f0b90a6ddac59cc5fd0e
Jul 28 19:47:57.135477 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.134767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq" event={"ID":"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6","Type":"ContainerStarted","Data":"f9832ecf1da2a5b92fcc9c6b9909120f09b14870589d69006c4cc627f5b291a9"}
Jul 28 19:47:57.135477 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.134849    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq" event={"ID":"9a3e3e17-12ba-421f-85a3-4a1d2f0d06a6","Type":"ContainerStarted","Data":"f93477546a4805e1f649f7cd4dbf4c43d6b927ad1df16046197bba4e15ce0759"}
Jul 28 19:47:57.136129 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.136092    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-kpzrq" event={"ID":"f18750e0-eeda-45c0-9150-8240779c4755","Type":"ContainerStarted","Data":"3cfb7e065e8880351b93c03e2d661e8f741d2eb2a685f0b90a6ddac59cc5fd0e"}
Jul 28 19:47:57.138098 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.138042    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sw5" event={"ID":"fec949f3-4c83-40dd-8b20-73eca8c3d8ef","Type":"ContainerStarted","Data":"4f3ba806376e7704bcdbff75fa4aca373b0b6ff1e6fed20cbf795947634c116e"}
Jul 28 19:47:57.139344 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.139321    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm" event={"ID":"3f309a2c-89e8-496a-bd16-2a7f1eb853e7","Type":"ContainerStarted","Data":"d89467f4393a57b5ef8204014ab77d6f72aa4db60d1218c39222ab61a449c5fe"}
Jul 28 19:47:57.140938 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.140916    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d" event={"ID":"f985925d-7690-4f3d-b33d-d12804131a78","Type":"ContainerStarted","Data":"9f4f0ee0c76dbc26f2b6519d395e20421341df5a40f558f4c8123cb5c6545498"}
Jul 28 19:47:57.142450 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.142421    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98f64cb9-j45cl" event={"ID":"56e2de65-a321-45dd-9ca8-bc13330c4a64","Type":"ContainerStarted","Data":"65a02f69b985c0986c9ea364060ae951a27f9a85a9f2e3d2710ae2d1fea55ec2"}
Jul 28 19:47:57.144024 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.144001    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt" event={"ID":"ad0030e8-c33f-49c3-bb42-d0b809f19f98","Type":"ContainerStarted","Data":"f67df300029e89c6e4f8a0b24b630f9ef06728872034d18ba116268124a9eb20"}
Jul 28 19:47:57.154913 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.154866    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-wq9qq" podStartSLOduration=33.998999582 podStartE2EDuration="39.154853739s" totalImagesPullingTime="5.155854157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:57.153932599 +0000 UTC m=+80.054149442" watchObservedRunningTime="2026-07-28 19:47:57.154853739 +0000 UTC m=+80.055070620"
Jul 28 19:47:57.172026 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.171972    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-j8m4d" podStartSLOduration=33.992534372 podStartE2EDuration="39.171954556s" totalImagesPullingTime="5.179420184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:57.170407778 +0000 UTC m=+80.070624621" watchObservedRunningTime="2026-07-28 19:47:57.171954556 +0000 UTC m=+80.072171397"
Jul 28 19:47:57.186252 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:47:57.186190    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-gsjpt" podStartSLOduration=34.960471018 podStartE2EDuration="39.186170853s" totalImagesPullingTime="4.225699835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:47:57.185619133 +0000 UTC m=+80.085836003" watchObservedRunningTime="2026-07-28 19:47:57.186170853 +0000 UTC m=+80.086387697"
Jul 28 19:48:01.115459 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:01.115337    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jh7xz"
Jul 28 19:48:05.202470 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.202096    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9sw5" event={"ID":"fec949f3-4c83-40dd-8b20-73eca8c3d8ef","Type":"ContainerStarted","Data":"0106ae79c6aa8dc9fd19ed615db8ef3db012e7492403d6b2a08d423438549b6c"}
Jul 28 19:48:05.204222 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.204187    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm" event={"ID":"3f309a2c-89e8-496a-bd16-2a7f1eb853e7","Type":"ContainerStarted","Data":"7231ee118a5fcad76bec2d77a5886821b80124d84270070f567f2d61c268c571"}
Jul 28 19:48:05.204391 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.204300    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:48:05.206918 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.206854    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98f64cb9-j45cl" event={"ID":"56e2de65-a321-45dd-9ca8-bc13330c4a64","Type":"ContainerStarted","Data":"d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01"}
Jul 28 19:48:05.207038 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.207023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm"
Jul 28 19:48:05.221645 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.221595    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l9sw5" podStartSLOduration=17.672620075 podStartE2EDuration="39.221579303s" totalImagesPullingTime="21.548959228s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 19:47:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:05.219783663 +0000 UTC m=+88.120000506" watchObservedRunningTime="2026-07-28 19:48:05.221579303 +0000 UTC m=+88.121796151"
Jul 28 19:48:05.237863 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.237581    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d98f64cb9-j45cl" podStartSLOduration=4.877640747 podStartE2EDuration="13.237561828s" totalImagesPullingTime="8.359921081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:05.235850488 +0000 UTC m=+88.136067331" watchObservedRunningTime="2026-07-28 19:48:05.237561828 +0000 UTC m=+88.137778678"
Jul 28 19:48:05.257154 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:05.257095    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58d7d84c75-cwzmm" podStartSLOduration=4.890530395 podStartE2EDuration="13.257077794s" totalImagesPullingTime="8.366547399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:05.25605542 +0000 UTC m=+88.156272275" watchObservedRunningTime="2026-07-28 19:48:05.257077794 +0000 UTC m=+88.157294751"
Jul 28 19:48:07.076793 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:07.076765    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:48:11.565143 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.565107    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-l7kqt"]
Jul 28 19:48:11.686454 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.686411    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.689753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.689485    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 19:48:11.689753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.689537    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 19:48:11.689753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.689557    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lrswp\""
Jul 28 19:48:11.689753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.689537    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 19:48:11.689753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.689567    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 19:48:11.715918 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.715888    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716091 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.715939    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddnpz\" (UniqueName: \"kubernetes.io/projected/fd064301-83bd-4224-a7ce-53552a9543fa-kube-api-access-ddnpz\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716091 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.715969    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-sys\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716091 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.715992    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-root\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716091 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.716014    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716091 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.716048    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd064301-83bd-4224-a7ce-53552a9543fa-metrics-client-ca\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716376 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.716129    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-wtmp\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716376 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.716190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-textfile\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.716376 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.716287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817418 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817328    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddnpz\" (UniqueName: \"kubernetes.io/projected/fd064301-83bd-4224-a7ce-53552a9543fa-kube-api-access-ddnpz\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817418 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817385    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-sys\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817418 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817412    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-root\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817468    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd064301-83bd-4224-a7ce-53552a9543fa-metrics-client-ca\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817487    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-wtmp\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817499    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-sys\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-textfile\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817587    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.817717 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817646    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.818052 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-textfile\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.818052 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.817898    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-root\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.818052 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:11.817964    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 19:48:11.818052 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:11.818044    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls podName:fd064301-83bd-4224-a7ce-53552a9543fa nodeName:}" failed. No retries permitted until 2026-07-28 19:48:12.317997759 +0000 UTC m=+95.218214579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls") pod "node-exporter-l7kqt" (UID: "fd064301-83bd-4224-a7ce-53552a9543fa") : secret "node-exporter-tls" not found
Jul 28 19:48:11.818298 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.818212    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-wtmp\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.818380 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.818354    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-accelerators-collector-config\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.818725 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.818705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fd064301-83bd-4224-a7ce-53552a9543fa-metrics-client-ca\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.820528 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.820506    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:11.829428 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:11.829399    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddnpz\" (UniqueName: \"kubernetes.io/projected/fd064301-83bd-4224-a7ce-53552a9543fa-kube-api-access-ddnpz\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:12.322110 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:12.322067    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:12.322442 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:12.322294    2578 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 19:48:12.322442 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:12.322379    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls podName:fd064301-83bd-4224-a7ce-53552a9543fa nodeName:}" failed. No retries permitted until 2026-07-28 19:48:13.322362181 +0000 UTC m=+96.222579004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls") pod "node-exporter-l7kqt" (UID: "fd064301-83bd-4224-a7ce-53552a9543fa") : secret "node-exporter-tls" not found
Jul 28 19:48:12.729326 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:12.729229    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d98f64cb9-j45cl"]
Jul 28 19:48:13.229979 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:13.229894    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:48:13.331074 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:13.331029    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:13.333736 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:13.333705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fd064301-83bd-4224-a7ce-53552a9543fa-node-exporter-tls\") pod \"node-exporter-l7kqt\" (UID: \"fd064301-83bd-4224-a7ce-53552a9543fa\") " pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:13.497528 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:13.497436    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-l7kqt"
Jul 28 19:48:14.596522 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:14.596488    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-698cf84dbb-dcxv5"]
Jul 28 19:48:15.058935 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:15.058899    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-n995h"
Jul 28 19:48:16.561234 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:48:16.561200    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd064301_83bd_4224_a7ce_53552a9543fa.slice/crio-bb1991203bcc53b8c14dcef17ed4da44cdf716b76e8db0b123cb1e9d18a81b12 WatchSource:0}: Error finding container bb1991203bcc53b8c14dcef17ed4da44cdf716b76e8db0b123cb1e9d18a81b12: Status 404 returned error can't find the container with id bb1991203bcc53b8c14dcef17ed4da44cdf716b76e8db0b123cb1e9d18a81b12
Jul 28 19:48:17.261029 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.260988    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7kqt" event={"ID":"fd064301-83bd-4224-a7ce-53552a9543fa","Type":"ContainerStarted","Data":"bb1991203bcc53b8c14dcef17ed4da44cdf716b76e8db0b123cb1e9d18a81b12"}
Jul 28 19:48:17.523928 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.523901    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bff96b74f-wk7lv"]
Jul 28 19:48:17.534064 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.534038    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.537319 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.537293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bff96b74f-wk7lv"]
Jul 28 19:48:17.548980 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.548942    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 19:48:17.571336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571289    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5fn\" (UniqueName: \"kubernetes.io/projected/4e2b8504-e4b5-43bd-adbd-c82de27900e3-kube-api-access-qm5fn\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.571336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571335    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-config\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.571795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-oauth-serving-cert\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.571795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-trusted-ca-bundle\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.571795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571489    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-serving-cert\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.571795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-oauth-config\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.571795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.571551    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-service-ca\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673088 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673040    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-oauth-serving-cert\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673088 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-trusted-ca-bundle\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673428 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673129    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-serving-cert\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673428 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-oauth-config\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673428 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673193    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-service-ca\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673428 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673241    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5fn\" (UniqueName: \"kubernetes.io/projected/4e2b8504-e4b5-43bd-adbd-c82de27900e3-kube-api-access-qm5fn\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.673428 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.673300    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-config\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.674055 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.674029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-config\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.674337 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.674302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-service-ca\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.674467 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.674303    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-trusted-ca-bundle\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.674878 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.674855    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-oauth-serving-cert\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.676154 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.676127    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-oauth-config\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.676261 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.676226    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-serving-cert\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.681907 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.681887    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5fn\" (UniqueName: \"kubernetes.io/projected/4e2b8504-e4b5-43bd-adbd-c82de27900e3-kube-api-access-qm5fn\") pod \"console-7bff96b74f-wk7lv\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") " pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:17.846615 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:17.846517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:18.057815 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:18.057723    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bff96b74f-wk7lv"]
Jul 28 19:48:18.253582 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:48:18.253497    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e2b8504_e4b5_43bd_adbd_c82de27900e3.slice/crio-cbbbe33ced2f39f4925ba81f221f09dde0e4513ec8321596d96137da16a19b23 WatchSource:0}: Error finding container cbbbe33ced2f39f4925ba81f221f09dde0e4513ec8321596d96137da16a19b23: Status 404 returned error can't find the container with id cbbbe33ced2f39f4925ba81f221f09dde0e4513ec8321596d96137da16a19b23
Jul 28 19:48:18.267648 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:18.267606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-kpzrq" event={"ID":"f18750e0-eeda-45c0-9150-8240779c4755","Type":"ContainerStarted","Data":"9fd186b802299230e0da59c299d347fe8f74d3785ac09128cf14a1002ea5279b"}
Jul 28 19:48:18.267990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:18.267928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:48:18.269735 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:18.269675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bff96b74f-wk7lv" event={"ID":"4e2b8504-e4b5-43bd-adbd-c82de27900e3","Type":"ContainerStarted","Data":"cbbbe33ced2f39f4925ba81f221f09dde0e4513ec8321596d96137da16a19b23"}
Jul 28 19:48:18.277256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:18.277230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-kpzrq"
Jul 28 19:48:18.308801 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:18.308750    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-kpzrq" podStartSLOduration=5.534214385 podStartE2EDuration="26.308726902s" totalImagesPullingTime="20.774512517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:18.307738898 +0000 UTC m=+101.207955756" watchObservedRunningTime="2026-07-28 19:48:18.308726902 +0000 UTC m=+101.208943748"
Jul 28 19:48:19.276909 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:19.276864    2578 generic.go:356] "Generic (PLEG): container finished" podID="fd064301-83bd-4224-a7ce-53552a9543fa" containerID="3eb57adad49bb9fede3429db48d74507d84f24257133b2f226c0244e6dc5299e" exitCode=0
Jul 28 19:48:19.277416 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:19.277037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7kqt" event={"ID":"fd064301-83bd-4224-a7ce-53552a9543fa","Type":"ContainerDied","Data":"3eb57adad49bb9fede3429db48d74507d84f24257133b2f226c0244e6dc5299e"}
Jul 28 19:48:19.279441 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:19.279411    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bff96b74f-wk7lv" event={"ID":"4e2b8504-e4b5-43bd-adbd-c82de27900e3","Type":"ContainerStarted","Data":"d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a"}
Jul 28 19:48:19.311715 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:19.311655    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bff96b74f-wk7lv" podStartSLOduration=2.311631356 podStartE2EDuration="2.311631356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:19.310339605 +0000 UTC m=+102.210556449" watchObservedRunningTime="2026-07-28 19:48:19.311631356 +0000 UTC m=+102.211848201"
Jul 28 19:48:20.287817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:20.287770    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7kqt" event={"ID":"fd064301-83bd-4224-a7ce-53552a9543fa","Type":"ContainerStarted","Data":"a6c38c1de020c7f08d6a80778e0dd58cbeee1af1e2264dee7ce65ac2a44baa64"}
Jul 28 19:48:20.288317 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:20.287844    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-l7kqt" event={"ID":"fd064301-83bd-4224-a7ce-53552a9543fa","Type":"ContainerStarted","Data":"6669bd6e7f797f359071a38dde23bb9ac2aab2f788fcf88b5a76d594a5e19801"}
Jul 28 19:48:20.307107 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:20.307049    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-l7kqt" podStartSLOduration=7.4798392830000004 podStartE2EDuration="9.307029341s" totalImagesPullingTime="1.715091023s" totalInitContainerRuntime="112.099035ms" isStatefulPod=true podCreationTimestamp="2026-07-28 19:48:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:20.305543821 +0000 UTC m=+103.205760677" watchObservedRunningTime="2026-07-28 19:48:20.307029341 +0000 UTC m=+103.207246187"
Jul 28 19:48:21.565631 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.565587    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bff96b74f-wk7lv"]
Jul 28 19:48:21.593439 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.593405    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58b4f54949-v29cp"]
Jul 28 19:48:21.640465 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.640416    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b4f54949-v29cp"]
Jul 28 19:48:21.640616 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.640526    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709032 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.708989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-oauth-serving-cert\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709218 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.709045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-oauth-config\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709218 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.709072    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-serving-cert\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709218 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.709148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-service-ca\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709218 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.709183    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbt8z\" (UniqueName: \"kubernetes.io/projected/1def900f-5e3d-4dc4-b9a3-07be9312650e-kube-api-access-wbt8z\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709488 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.709291    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-trusted-ca-bundle\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.709488 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.709363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-config\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810129 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-config\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810129 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-oauth-serving-cert\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810388 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810260    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-oauth-config\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810388 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-serving-cert\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810507 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-service-ca\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810507 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810426    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbt8z\" (UniqueName: \"kubernetes.io/projected/1def900f-5e3d-4dc4-b9a3-07be9312650e-kube-api-access-wbt8z\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.810507 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.810492    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-trusted-ca-bundle\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.811025 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.811000    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-oauth-serving-cert\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.811429 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.811404    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-config\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.812024 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.811994    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-service-ca\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.812130 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.812070    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-trusted-ca-bundle\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.813234 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.813209    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-oauth-config\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.813395 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.813374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-serving-cert\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.818971 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.818912    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbt8z\" (UniqueName: \"kubernetes.io/projected/1def900f-5e3d-4dc4-b9a3-07be9312650e-kube-api-access-wbt8z\") pod \"console-58b4f54949-v29cp\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") " pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:21.952360 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:21.952322    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:22.159188 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:22.159155    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58b4f54949-v29cp"]
Jul 28 19:48:22.162148 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:48:22.162114    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1def900f_5e3d_4dc4_b9a3_07be9312650e.slice/crio-0d9ddbd5c51d84d5ac8742656498e3eb2b469b981fce18fca393460f85c6b30b WatchSource:0}: Error finding container 0d9ddbd5c51d84d5ac8742656498e3eb2b469b981fce18fca393460f85c6b30b: Status 404 returned error can't find the container with id 0d9ddbd5c51d84d5ac8742656498e3eb2b469b981fce18fca393460f85c6b30b
Jul 28 19:48:22.300609 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:22.300570    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4f54949-v29cp" event={"ID":"1def900f-5e3d-4dc4-b9a3-07be9312650e","Type":"ContainerStarted","Data":"765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4"}
Jul 28 19:48:22.300785 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:22.300628    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4f54949-v29cp" event={"ID":"1def900f-5e3d-4dc4-b9a3-07be9312650e","Type":"ContainerStarted","Data":"0d9ddbd5c51d84d5ac8742656498e3eb2b469b981fce18fca393460f85c6b30b"}
Jul 28 19:48:22.317711 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:22.317662    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58b4f54949-v29cp" podStartSLOduration=1.3176463090000001 podStartE2EDuration="1.317646309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:48:22.316585975 +0000 UTC m=+105.216802817" watchObservedRunningTime="2026-07-28 19:48:22.317646309 +0000 UTC m=+105.217863151"
Jul 28 19:48:27.846636 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:27.846590    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:31.952631 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:31.952594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:31.953143 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:31.952670    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:31.957378 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:31.957352    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:32.343784 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:32.343751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:48:37.360280 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:37.360242    2578 generic.go:356] "Generic (PLEG): container finished" podID="9d48cb85-c537-4a11-90f7-1da5815cc0ba" containerID="f3a90f9318253b98ce5f0e0ab3b9ae9686d69c7ef135107ceca3feaf16974038" exitCode=0
Jul 28 19:48:37.360721 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:37.360325    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf" event={"ID":"9d48cb85-c537-4a11-90f7-1da5815cc0ba","Type":"ContainerDied","Data":"f3a90f9318253b98ce5f0e0ab3b9ae9686d69c7ef135107ceca3feaf16974038"}
Jul 28 19:48:37.360721 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:37.360679    2578 scope.go:117] "RemoveContainer" containerID="f3a90f9318253b98ce5f0e0ab3b9ae9686d69c7ef135107ceca3feaf16974038"
Jul 28 19:48:37.753320 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:37.753213    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d98f64cb9-j45cl" podUID="56e2de65-a321-45dd-9ca8-bc13330c4a64" containerName="console" containerID="cri-o://d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01" gracePeriod=15
Jul 28 19:48:38.033891 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.033863    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:48:38.160483 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.160437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-oauth-config\") pod \"56e2de65-a321-45dd-9ca8-bc13330c4a64\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") "
Jul 28 19:48:38.160682 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.160493    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9vsf\" (UniqueName: \"kubernetes.io/projected/56e2de65-a321-45dd-9ca8-bc13330c4a64-kube-api-access-k9vsf\") pod \"56e2de65-a321-45dd-9ca8-bc13330c4a64\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") "
Jul 28 19:48:38.160682 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.160532    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-serving-cert\") pod \"56e2de65-a321-45dd-9ca8-bc13330c4a64\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") "
Jul 28 19:48:38.160795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.160686    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-config\") pod \"56e2de65-a321-45dd-9ca8-bc13330c4a64\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") "
Jul 28 19:48:38.160795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.160766    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-oauth-serving-cert\") pod \"56e2de65-a321-45dd-9ca8-bc13330c4a64\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") "
Jul 28 19:48:38.160953 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.160808    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-service-ca\") pod \"56e2de65-a321-45dd-9ca8-bc13330c4a64\" (UID: \"56e2de65-a321-45dd-9ca8-bc13330c4a64\") "
Jul 28 19:48:38.161124 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.161088    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-config" (OuterVolumeSpecName: "console-config") pod "56e2de65-a321-45dd-9ca8-bc13330c4a64" (UID: "56e2de65-a321-45dd-9ca8-bc13330c4a64"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:38.161124 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.161098    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "56e2de65-a321-45dd-9ca8-bc13330c4a64" (UID: "56e2de65-a321-45dd-9ca8-bc13330c4a64"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:38.161503 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.161141    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-service-ca" (OuterVolumeSpecName: "service-ca") pod "56e2de65-a321-45dd-9ca8-bc13330c4a64" (UID: "56e2de65-a321-45dd-9ca8-bc13330c4a64"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:38.162953 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.162927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "56e2de65-a321-45dd-9ca8-bc13330c4a64" (UID: "56e2de65-a321-45dd-9ca8-bc13330c4a64"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:48:38.163048 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.162969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "56e2de65-a321-45dd-9ca8-bc13330c4a64" (UID: "56e2de65-a321-45dd-9ca8-bc13330c4a64"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:48:38.163048 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.162994    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e2de65-a321-45dd-9ca8-bc13330c4a64-kube-api-access-k9vsf" (OuterVolumeSpecName: "kube-api-access-k9vsf") pod "56e2de65-a321-45dd-9ca8-bc13330c4a64" (UID: "56e2de65-a321-45dd-9ca8-bc13330c4a64"). InnerVolumeSpecName "kube-api-access-k9vsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:48:38.261981 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.261940    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:38.261981 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.261972    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-oauth-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:38.261981 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.261984    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/56e2de65-a321-45dd-9ca8-bc13330c4a64-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:38.262213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.261997    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-oauth-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:38.262213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.262006    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vsf\" (UniqueName: \"kubernetes.io/projected/56e2de65-a321-45dd-9ca8-bc13330c4a64-kube-api-access-k9vsf\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:38.262213 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.262015    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/56e2de65-a321-45dd-9ca8-bc13330c4a64-console-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:38.366426 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.366332    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lbkmf" event={"ID":"9d48cb85-c537-4a11-90f7-1da5815cc0ba","Type":"ContainerStarted","Data":"5fbc175a623d79a8bd7d1e2560eb03b03e1ec8fbb628629c13f40d01b668db66"}
Jul 28 19:48:38.367513 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.367489    2578 generic.go:356] "Generic (PLEG): container finished" podID="56e2de65-a321-45dd-9ca8-bc13330c4a64" containerID="d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01" exitCode=0
Jul 28 19:48:38.367634 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.367528    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d98f64cb9-j45cl"
Jul 28 19:48:38.367634 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.367579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98f64cb9-j45cl" event={"ID":"56e2de65-a321-45dd-9ca8-bc13330c4a64","Type":"ContainerDied","Data":"d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01"}
Jul 28 19:48:38.367743 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.367635    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d98f64cb9-j45cl" event={"ID":"56e2de65-a321-45dd-9ca8-bc13330c4a64","Type":"ContainerDied","Data":"65a02f69b985c0986c9ea364060ae951a27f9a85a9f2e3d2710ae2d1fea55ec2"}
Jul 28 19:48:38.367743 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.367664    2578 scope.go:117] "RemoveContainer" containerID="d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01"
Jul 28 19:48:38.377036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.377019    2578 scope.go:117] "RemoveContainer" containerID="d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01"
Jul 28 19:48:38.377312 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:38.377291    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01\": container with ID starting with d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01 not found: ID does not exist" containerID="d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01"
Jul 28 19:48:38.377380 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.377324    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01"} err="failed to get container status \"d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01\": rpc error: code = NotFound desc = could not find container \"d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01\": container with ID starting with d338854e2899e9832f09107dd7f9f6930854bf88f46465dabf6e3bfc8270de01 not found: ID does not exist"
Jul 28 19:48:38.396482 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.396450    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d98f64cb9-j45cl"]
Jul 28 19:48:38.402792 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:38.402759    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d98f64cb9-j45cl"]
Jul 28 19:48:39.625183 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:39.625142    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5" podUID="34065413-44cf-4622-9f6f-be947f1d7c82" containerName="registry" containerID="cri-o://dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed" gracePeriod=30
Jul 28 19:48:39.650780 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:39.650740    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56e2de65-a321-45dd-9ca8-bc13330c4a64" path="/var/lib/kubelet/pods/56e2de65-a321-45dd-9ca8-bc13330c4a64/volumes"
Jul 28 19:48:39.902361 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:39.902299    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:48:40.079854 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.079820    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34065413-44cf-4622-9f6f-be947f1d7c82-ca-trust-extracted\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080042 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.079859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-bound-sa-token\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080042 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.079898    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-installation-pull-secrets\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080042 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.079956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l9zff\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-kube-api-access-l9zff\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080042 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.079985    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-image-registry-private-configuration\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080042 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.080020    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-trusted-ca\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.080071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-registry-certificates\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.080095    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") pod \"34065413-44cf-4622-9f6f-be947f1d7c82\" (UID: \"34065413-44cf-4622-9f6f-be947f1d7c82\") "
Jul 28 19:48:40.080685 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.080635    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:40.080809 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.080768    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:40.083094 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.083064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:48:40.083190 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.083136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:48:40.083253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.083189    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:48:40.083253 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.083240    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:48:40.083395 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.083297    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-kube-api-access-l9zff" (OuterVolumeSpecName: "kube-api-access-l9zff") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "kube-api-access-l9zff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:48:40.088829 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.088794    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34065413-44cf-4622-9f6f-be947f1d7c82-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "34065413-44cf-4622-9f6f-be947f1d7c82" (UID: "34065413-44cf-4622-9f6f-be947f1d7c82"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181232    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-trusted-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181285    2578 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/34065413-44cf-4622-9f6f-be947f1d7c82-registry-certificates\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181296    2578 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-registry-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181305    2578 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/34065413-44cf-4622-9f6f-be947f1d7c82-ca-trust-extracted\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181313    2578 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-bound-sa-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181321    2578 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-installation-pull-secrets\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181329    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9zff\" (UniqueName: \"kubernetes.io/projected/34065413-44cf-4622-9f6f-be947f1d7c82-kube-api-access-l9zff\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.181342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.181339    2578 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/34065413-44cf-4622-9f6f-be947f1d7c82-image-registry-private-configuration\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:40.377406 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.377371    2578 generic.go:356] "Generic (PLEG): container finished" podID="34065413-44cf-4622-9f6f-be947f1d7c82" containerID="dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed" exitCode=0
Jul 28 19:48:40.377572 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.377421    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5"
Jul 28 19:48:40.377572 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.377456    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5" event={"ID":"34065413-44cf-4622-9f6f-be947f1d7c82","Type":"ContainerDied","Data":"dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed"}
Jul 28 19:48:40.377572 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.377512    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-698cf84dbb-dcxv5" event={"ID":"34065413-44cf-4622-9f6f-be947f1d7c82","Type":"ContainerDied","Data":"dc9e583fd92130a6505675018a91a229a64187a625bb1c6b0c97c13d8c7d4bc9"}
Jul 28 19:48:40.377572 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.377534    2578 scope.go:117] "RemoveContainer" containerID="dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed"
Jul 28 19:48:40.386590 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.386572    2578 scope.go:117] "RemoveContainer" containerID="dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed"
Jul 28 19:48:40.386900 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:40.386881    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed\": container with ID starting with dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed not found: ID does not exist" containerID="dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed"
Jul 28 19:48:40.386950 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.386908    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed"} err="failed to get container status \"dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed\": rpc error: code = NotFound desc = could not find container \"dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed\": container with ID starting with dbbde1d20ea7cbc23abece9d4e740c36f610ce65c4b4ba91da7c255953c37bed not found: ID does not exist"
Jul 28 19:48:40.398470 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.398447    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-698cf84dbb-dcxv5"]
Jul 28 19:48:40.402194 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:40.402177    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-698cf84dbb-dcxv5"]
Jul 28 19:48:41.650743 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:41.650708    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34065413-44cf-4622-9f6f-be947f1d7c82" path="/var/lib/kubelet/pods/34065413-44cf-4622-9f6f-be947f1d7c82/volumes"
Jul 28 19:48:46.588873 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.588824    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bff96b74f-wk7lv" podUID="4e2b8504-e4b5-43bd-adbd-c82de27900e3" containerName="console" containerID="cri-o://d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a" gracePeriod=15
Jul 28 19:48:46.839043 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.838981    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:46.935502 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935461    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm5fn\" (UniqueName: \"kubernetes.io/projected/4e2b8504-e4b5-43bd-adbd-c82de27900e3-kube-api-access-qm5fn\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935502 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935503    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-trusted-ca-bundle\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935755 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935557    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-oauth-serving-cert\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935755 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935576    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-config\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935755 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-service-ca\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935755 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935618    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-oauth-config\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935755 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935647    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-serving-cert\") pod \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\" (UID: \"4e2b8504-e4b5-43bd-adbd-c82de27900e3\") "
Jul 28 19:48:46.935998 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935975    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-config" (OuterVolumeSpecName: "console-config") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:46.935998 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935982    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:46.936087 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.935998    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-service-ca" (OuterVolumeSpecName: "service-ca") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:46.936087 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.936013    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:48:46.937769 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.937736    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:48:46.937874 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.937777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e2b8504-e4b5-43bd-adbd-c82de27900e3-kube-api-access-qm5fn" (OuterVolumeSpecName: "kube-api-access-qm5fn") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "kube-api-access-qm5fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:48:46.937874 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:46.937807    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "4e2b8504-e4b5-43bd-adbd-c82de27900e3" (UID: "4e2b8504-e4b5-43bd-adbd-c82de27900e3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:48:47.036094 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036058    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-oauth-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.036094 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036088    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.036094 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036098    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.036363 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036106    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-oauth-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.036363 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036116    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e2b8504-e4b5-43bd-adbd-c82de27900e3-console-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.036363 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036124    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm5fn\" (UniqueName: \"kubernetes.io/projected/4e2b8504-e4b5-43bd-adbd-c82de27900e3-kube-api-access-qm5fn\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.036363 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.036133    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4e2b8504-e4b5-43bd-adbd-c82de27900e3-trusted-ca-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:48:47.407614 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.407576    2578 generic.go:356] "Generic (PLEG): container finished" podID="0377c7ec-588d-42a2-aed1-e21b9f626411" containerID="f9da7ffa2dbd44b034ebaedd615ea09f9a0029eb8cda6802baf0c101cc1965e1" exitCode=0
Jul 28 19:48:47.407795 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.407649    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2" event={"ID":"0377c7ec-588d-42a2-aed1-e21b9f626411","Type":"ContainerDied","Data":"f9da7ffa2dbd44b034ebaedd615ea09f9a0029eb8cda6802baf0c101cc1965e1"}
Jul 28 19:48:47.407997 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.407980    2578 scope.go:117] "RemoveContainer" containerID="f9da7ffa2dbd44b034ebaedd615ea09f9a0029eb8cda6802baf0c101cc1965e1"
Jul 28 19:48:47.408925 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.408906    2578 generic.go:356] "Generic (PLEG): container finished" podID="4e2b8504-e4b5-43bd-adbd-c82de27900e3" containerID="d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a" exitCode=0
Jul 28 19:48:47.409016 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.408974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bff96b74f-wk7lv"
Jul 28 19:48:47.409016 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.408987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bff96b74f-wk7lv" event={"ID":"4e2b8504-e4b5-43bd-adbd-c82de27900e3","Type":"ContainerDied","Data":"d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a"}
Jul 28 19:48:47.409123 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.409037    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bff96b74f-wk7lv" event={"ID":"4e2b8504-e4b5-43bd-adbd-c82de27900e3","Type":"ContainerDied","Data":"cbbbe33ced2f39f4925ba81f221f09dde0e4513ec8321596d96137da16a19b23"}
Jul 28 19:48:47.409123 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.409066    2578 scope.go:117] "RemoveContainer" containerID="d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a"
Jul 28 19:48:47.422298 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.422253    2578 scope.go:117] "RemoveContainer" containerID="d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a"
Jul 28 19:48:47.422628 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:48:47.422602    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a\": container with ID starting with d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a not found: ID does not exist" containerID="d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a"
Jul 28 19:48:47.422768 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.422638    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a"} err="failed to get container status \"d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a\": rpc error: code = NotFound desc = could not find container \"d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a\": container with ID starting with d2ef10b296d7117bbbeca530586ca611d403691f1cfff33cff546357380ef75a not found: ID does not exist"
Jul 28 19:48:47.438064 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.438041    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bff96b74f-wk7lv"]
Jul 28 19:48:47.443552 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.443530    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bff96b74f-wk7lv"]
Jul 28 19:48:47.650320 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:47.650252    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e2b8504-e4b5-43bd-adbd-c82de27900e3" path="/var/lib/kubelet/pods/4e2b8504-e4b5-43bd-adbd-c82de27900e3/volumes"
Jul 28 19:48:48.414952 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:48.414909    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2" event={"ID":"0377c7ec-588d-42a2-aed1-e21b9f626411","Type":"ContainerStarted","Data":"927ca61e81b28ded8574b1f73f72c2de3607bf6438f943458c57edfa2280a384"}
Jul 28 19:48:57.449901 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:57.449862    2578 generic.go:356] "Generic (PLEG): container finished" podID="3adb22c5-5525-40b6-a40c-42a13c6abd3a" containerID="45b6fc466cca940d7fac7bef75a39d093f5dff4b1c0a129ecf1cbb15c085a6b5" exitCode=0
Jul 28 19:48:57.450437 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:57.449924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j66nk" event={"ID":"3adb22c5-5525-40b6-a40c-42a13c6abd3a","Type":"ContainerDied","Data":"45b6fc466cca940d7fac7bef75a39d093f5dff4b1c0a129ecf1cbb15c085a6b5"}
Jul 28 19:48:57.450437 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:57.450345    2578 scope.go:117] "RemoveContainer" containerID="45b6fc466cca940d7fac7bef75a39d093f5dff4b1c0a129ecf1cbb15c085a6b5"
Jul 28 19:48:58.454967 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:48:58.454926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-j66nk" event={"ID":"3adb22c5-5525-40b6-a40c-42a13c6abd3a","Type":"ContainerStarted","Data":"dfc8409fdea079f2bb7d6efb9cccd6781dff01a21bf0290ed644a7ff21c69429"}
Jul 28 19:49:42.411609 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411572    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbc95699-sbbrr"]
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411872    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34065413-44cf-4622-9f6f-be947f1d7c82" containerName="registry"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411884    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="34065413-44cf-4622-9f6f-be947f1d7c82" containerName="registry"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411905    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e2b8504-e4b5-43bd-adbd-c82de27900e3" containerName="console"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411910    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e2b8504-e4b5-43bd-adbd-c82de27900e3" containerName="console"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411918    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56e2de65-a321-45dd-9ca8-bc13330c4a64" containerName="console"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411923    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="56e2de65-a321-45dd-9ca8-bc13330c4a64" containerName="console"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411961    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="56e2de65-a321-45dd-9ca8-bc13330c4a64" containerName="console"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411969    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e2b8504-e4b5-43bd-adbd-c82de27900e3" containerName="console"
Jul 28 19:49:42.412006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.411990    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="34065413-44cf-4622-9f6f-be947f1d7c82" containerName="registry"
Jul 28 19:49:42.414755 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.414725    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.425368 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.425340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbc95699-sbbrr"]
Jul 28 19:49:42.577470 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577435    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-trusted-ca-bundle\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.577470 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlmf5\" (UniqueName: \"kubernetes.io/projected/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-kube-api-access-dlmf5\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.577688 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-oauth-config\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.577688 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577544    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-config\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.577688 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577585    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-service-ca\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.577688 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577633    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-serving-cert\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.577817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.577696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-oauth-serving-cert\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679161 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679062    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-oauth-serving-cert\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679161 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679127    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-trusted-ca-bundle\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlmf5\" (UniqueName: \"kubernetes.io/projected/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-kube-api-access-dlmf5\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679185    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-oauth-config\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679206    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-config\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-service-ca\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-serving-cert\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.679934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679902    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-oauth-serving-cert\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.680198 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-service-ca\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.680198 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.679995    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-config\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.680342 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.680206    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-trusted-ca-bundle\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.681792 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.681758    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-oauth-config\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.681921 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.681899    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-serving-cert\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.686951 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.686933    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlmf5\" (UniqueName: \"kubernetes.io/projected/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-kube-api-access-dlmf5\") pod \"console-6fbc95699-sbbrr\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") " pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.725536 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.725505    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:42.876594 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:42.876559    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbc95699-sbbrr"]
Jul 28 19:49:42.880011 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:49:42.879980    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e368a7_cc13_43f0_b3b5_de7f65b2ea7b.slice/crio-21f399ef6dc8591cefab270d00c285830a01b545c63e8da31f0d20a437176b0b WatchSource:0}: Error finding container 21f399ef6dc8591cefab270d00c285830a01b545c63e8da31f0d20a437176b0b: Status 404 returned error can't find the container with id 21f399ef6dc8591cefab270d00c285830a01b545c63e8da31f0d20a437176b0b
Jul 28 19:49:43.630729 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:43.630688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbc95699-sbbrr" event={"ID":"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b","Type":"ContainerStarted","Data":"abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f"}
Jul 28 19:49:43.631124 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:43.630742    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbc95699-sbbrr" event={"ID":"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b","Type":"ContainerStarted","Data":"21f399ef6dc8591cefab270d00c285830a01b545c63e8da31f0d20a437176b0b"}
Jul 28 19:49:43.647217 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:43.647172    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fbc95699-sbbrr" podStartSLOduration=1.647155505 podStartE2EDuration="1.647155505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:49:43.647109155 +0000 UTC m=+186.547325997" watchObservedRunningTime="2026-07-28 19:49:43.647155505 +0000 UTC m=+186.547372347"
Jul 28 19:49:52.726358 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:52.726317    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:52.726792 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:52.726432    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:52.731531 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:52.731511    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:53.673770 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:53.673739    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:49:53.716453 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:49:53.716411    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b4f54949-v29cp"]
Jul 28 19:50:18.737893 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:18.737834    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58b4f54949-v29cp" podUID="1def900f-5e3d-4dc4-b9a3-07be9312650e" containerName="console" containerID="cri-o://765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4" gracePeriod=15
Jul 28 19:50:18.973713 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:18.973688    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:50:19.069593 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-serving-cert\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.069593 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069598    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-trusted-ca-bundle\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.069847 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069642    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-config\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.069847 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069759    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbt8z\" (UniqueName: \"kubernetes.io/projected/1def900f-5e3d-4dc4-b9a3-07be9312650e-kube-api-access-wbt8z\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.069847 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069824    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-oauth-serving-cert\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.070008 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069858    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-service-ca\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.070008 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.069902    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-oauth-config\") pod \"1def900f-5e3d-4dc4-b9a3-07be9312650e\" (UID: \"1def900f-5e3d-4dc4-b9a3-07be9312650e\") "
Jul 28 19:50:19.070140 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.070106    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:50:19.070202 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.070170    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-config" (OuterVolumeSpecName: "console-config") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:50:19.070202 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.070188    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:50:19.070298 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.070213    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-service-ca" (OuterVolumeSpecName: "service-ca") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:50:19.071926 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.071894    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:50:19.072081 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.072062    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1def900f-5e3d-4dc4-b9a3-07be9312650e-kube-api-access-wbt8z" (OuterVolumeSpecName: "kube-api-access-wbt8z") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "kube-api-access-wbt8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:50:19.072123 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.072090    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1def900f-5e3d-4dc4-b9a3-07be9312650e" (UID: "1def900f-5e3d-4dc4-b9a3-07be9312650e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:50:19.170898 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170855    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.170898 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170887    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbt8z\" (UniqueName: \"kubernetes.io/projected/1def900f-5e3d-4dc4-b9a3-07be9312650e-kube-api-access-wbt8z\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.170898 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170897    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-oauth-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.170898 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170906    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.171158 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170916    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-oauth-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.171158 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170926    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1def900f-5e3d-4dc4-b9a3-07be9312650e-console-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.171158 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.170934    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1def900f-5e3d-4dc4-b9a3-07be9312650e-trusted-ca-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:19.766218 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.766181    2578 generic.go:356] "Generic (PLEG): container finished" podID="1def900f-5e3d-4dc4-b9a3-07be9312650e" containerID="765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4" exitCode=0
Jul 28 19:50:19.766674 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.766223    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58b4f54949-v29cp"
Jul 28 19:50:19.766674 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.766289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4f54949-v29cp" event={"ID":"1def900f-5e3d-4dc4-b9a3-07be9312650e","Type":"ContainerDied","Data":"765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4"}
Jul 28 19:50:19.766674 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.766336    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58b4f54949-v29cp" event={"ID":"1def900f-5e3d-4dc4-b9a3-07be9312650e","Type":"ContainerDied","Data":"0d9ddbd5c51d84d5ac8742656498e3eb2b469b981fce18fca393460f85c6b30b"}
Jul 28 19:50:19.766674 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.766358    2578 scope.go:117] "RemoveContainer" containerID="765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4"
Jul 28 19:50:19.776260 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.776240    2578 scope.go:117] "RemoveContainer" containerID="765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4"
Jul 28 19:50:19.776593 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:50:19.776563    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4\": container with ID starting with 765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4 not found: ID does not exist" containerID="765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4"
Jul 28 19:50:19.776711 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.776603    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4"} err="failed to get container status \"765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4\": rpc error: code = NotFound desc = could not find container \"765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4\": container with ID starting with 765241733d2be83037b18ece3c5bb1bf5cc6d3ed39c617caacd66a9726b654b4 not found: ID does not exist"
Jul 28 19:50:19.784019 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.783996    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58b4f54949-v29cp"]
Jul 28 19:50:19.788167 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:19.788140    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58b4f54949-v29cp"]
Jul 28 19:50:21.649671 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:21.649634    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1def900f-5e3d-4dc4-b9a3-07be9312650e" path="/var/lib/kubelet/pods/1def900f-5e3d-4dc4-b9a3-07be9312650e/volumes"
Jul 28 19:50:22.210765 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.210728    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"]
Jul 28 19:50:22.211022 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.211011    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1def900f-5e3d-4dc4-b9a3-07be9312650e" containerName="console"
Jul 28 19:50:22.211070 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.211024    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1def900f-5e3d-4dc4-b9a3-07be9312650e" containerName="console"
Jul 28 19:50:22.211107 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.211080    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1def900f-5e3d-4dc4-b9a3-07be9312650e" containerName="console"
Jul 28 19:50:22.214003 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.213984    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.216698 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.216670    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 19:50:22.216698 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.216683    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 19:50:22.216855 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.216672    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ssthg\""
Jul 28 19:50:22.222541 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.222507    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"]
Jul 28 19:50:22.296059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.296021    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq5ws\" (UniqueName: \"kubernetes.io/projected/7fae94d2-cb97-4961-844b-62f5da91bd22-kube-api-access-qq5ws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.296059 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.296076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.296331 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.296148    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.397249 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.397212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq5ws\" (UniqueName: \"kubernetes.io/projected/7fae94d2-cb97-4961-844b-62f5da91bd22-kube-api-access-qq5ws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.397389 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.397290    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.397389 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.397339    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.397703 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.397682    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.397747 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.397690    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.405872 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.405835    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq5ws\" (UniqueName: \"kubernetes.io/projected/7fae94d2-cb97-4961-844b-62f5da91bd22-kube-api-access-qq5ws\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.523080 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.523045    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:22.655419 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.655323    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"]
Jul 28 19:50:22.658462 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:50:22.658432    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fae94d2_cb97_4961_844b_62f5da91bd22.slice/crio-d163bee931dc3b201e9e5eb565e28b6738abc2253608e224e118bfee6f33d02f WatchSource:0}: Error finding container d163bee931dc3b201e9e5eb565e28b6738abc2253608e224e118bfee6f33d02f: Status 404 returned error can't find the container with id d163bee931dc3b201e9e5eb565e28b6738abc2253608e224e118bfee6f33d02f
Jul 28 19:50:22.779207 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:22.779117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" event={"ID":"7fae94d2-cb97-4961-844b-62f5da91bd22","Type":"ContainerStarted","Data":"d163bee931dc3b201e9e5eb565e28b6738abc2253608e224e118bfee6f33d02f"}
Jul 28 19:50:30.813678 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:30.813639    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerID="89d8a50cff2c8cd4d393e5ab235aeb197af8e99603476b1eb3bf8e463afe763c" exitCode=0
Jul 28 19:50:30.814062 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:30.813688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" event={"ID":"7fae94d2-cb97-4961-844b-62f5da91bd22","Type":"ContainerDied","Data":"89d8a50cff2c8cd4d393e5ab235aeb197af8e99603476b1eb3bf8e463afe763c"}
Jul 28 19:50:32.824219 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:32.824186    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerID="813d67e877a05fa9abd27553cb1a4dc25bd1859d23f3ea35dfb06fd4da7e1599" exitCode=0
Jul 28 19:50:32.824600 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:32.824234    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" event={"ID":"7fae94d2-cb97-4961-844b-62f5da91bd22","Type":"ContainerDied","Data":"813d67e877a05fa9abd27553cb1a4dc25bd1859d23f3ea35dfb06fd4da7e1599"}
Jul 28 19:50:39.861873 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:39.861842    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" event={"ID":"7fae94d2-cb97-4961-844b-62f5da91bd22","Type":"ContainerStarted","Data":"111581894c58ae1c3d82d7879364368a1a728a3d985f15156242efc03ede1956"}
Jul 28 19:50:39.878816 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:39.878758    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" podStartSLOduration=1.8264666109999999 podStartE2EDuration="17.878741777s" totalImagesPullingTime="16.052275166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:50:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:50:39.877566632 +0000 UTC m=+242.777783500" watchObservedRunningTime="2026-07-28 19:50:39.878741777 +0000 UTC m=+242.778958618"
Jul 28 19:50:40.867227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:40.867189    2578 generic.go:356] "Generic (PLEG): container finished" podID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerID="111581894c58ae1c3d82d7879364368a1a728a3d985f15156242efc03ede1956" exitCode=0
Jul 28 19:50:40.867659 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:40.867315    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" event={"ID":"7fae94d2-cb97-4961-844b-62f5da91bd22","Type":"ContainerDied","Data":"111581894c58ae1c3d82d7879364368a1a728a3d985f15156242efc03ede1956"}
Jul 28 19:50:41.994386 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:41.994357    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:42.169709 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.169620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-bundle\") pod \"7fae94d2-cb97-4961-844b-62f5da91bd22\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") "
Jul 28 19:50:42.169884 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.169745    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-util\") pod \"7fae94d2-cb97-4961-844b-62f5da91bd22\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") "
Jul 28 19:50:42.169884 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.169786    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qq5ws\" (UniqueName: \"kubernetes.io/projected/7fae94d2-cb97-4961-844b-62f5da91bd22-kube-api-access-qq5ws\") pod \"7fae94d2-cb97-4961-844b-62f5da91bd22\" (UID: \"7fae94d2-cb97-4961-844b-62f5da91bd22\") "
Jul 28 19:50:42.170376 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.170344    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-bundle" (OuterVolumeSpecName: "bundle") pod "7fae94d2-cb97-4961-844b-62f5da91bd22" (UID: "7fae94d2-cb97-4961-844b-62f5da91bd22"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:50:42.172219 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.172192    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fae94d2-cb97-4961-844b-62f5da91bd22-kube-api-access-qq5ws" (OuterVolumeSpecName: "kube-api-access-qq5ws") pod "7fae94d2-cb97-4961-844b-62f5da91bd22" (UID: "7fae94d2-cb97-4961-844b-62f5da91bd22"). InnerVolumeSpecName "kube-api-access-qq5ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:50:42.175749 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.175725    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-util" (OuterVolumeSpecName: "util") pod "7fae94d2-cb97-4961-844b-62f5da91bd22" (UID: "7fae94d2-cb97-4961-844b-62f5da91bd22"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:50:42.271348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.271307    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:42.271348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.271343    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq5ws\" (UniqueName: \"kubernetes.io/projected/7fae94d2-cb97-4961-844b-62f5da91bd22-kube-api-access-qq5ws\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:42.271348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.271353    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7fae94d2-cb97-4961-844b-62f5da91bd22-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:50:42.877686 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.877660    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn"
Jul 28 19:50:42.877865 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.877661    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e675406xtsn" event={"ID":"7fae94d2-cb97-4961-844b-62f5da91bd22","Type":"ContainerDied","Data":"d163bee931dc3b201e9e5eb565e28b6738abc2253608e224e118bfee6f33d02f"}
Jul 28 19:50:42.877865 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:42.877776    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d163bee931dc3b201e9e5eb565e28b6738abc2253608e224e118bfee6f33d02f"
Jul 28 19:50:54.776787 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.776745    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"]
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777080    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="util"
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777095    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="util"
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777103    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="pull"
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777108    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="pull"
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777114    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="extract"
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777119    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="extract"
Jul 28 19:50:54.777407 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.777190    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fae94d2-cb97-4961-844b-62f5da91bd22" containerName="extract"
Jul 28 19:50:54.780612 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.780594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.783525 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.783489    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 19:50:54.783666 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.783586    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ssthg\""
Jul 28 19:50:54.784633 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.784615    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 19:50:54.815653 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.815617    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"]
Jul 28 19:50:54.869100 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.869052    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b8r\" (UniqueName: \"kubernetes.io/projected/9ea6686d-af93-4f4d-9c5d-7472ea97563f-kube-api-access-72b8r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.869100 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.869101    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.869383 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.869199    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.969985 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.969950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72b8r\" (UniqueName: \"kubernetes.io/projected/9ea6686d-af93-4f4d-9c5d-7472ea97563f-kube-api-access-72b8r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.970159 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.969993    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.970159 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.970047    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.970392 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.970376    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.970464 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.970447    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:54.978499 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:54.978477    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b8r\" (UniqueName: \"kubernetes.io/projected/9ea6686d-af93-4f4d-9c5d-7472ea97563f-kube-api-access-72b8r\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:55.089661 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:55.089564    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:50:55.224018 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:55.223987    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"]
Jul 28 19:50:55.227028 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:50:55.227000    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea6686d_af93_4f4d_9c5d_7472ea97563f.slice/crio-d1091720163c5d072edcb7f11919d7d1e6b264c90db322b0b2f9885be09e4889 WatchSource:0}: Error finding container d1091720163c5d072edcb7f11919d7d1e6b264c90db322b0b2f9885be09e4889: Status 404 returned error can't find the container with id d1091720163c5d072edcb7f11919d7d1e6b264c90db322b0b2f9885be09e4889
Jul 28 19:50:55.930022 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:55.929982    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerID="fef570eb96743881f6c50086e10a9ad34baf666db5432c79760e2ba6b2b3476b" exitCode=0
Jul 28 19:50:55.930022 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:55.930016    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp" event={"ID":"9ea6686d-af93-4f4d-9c5d-7472ea97563f","Type":"ContainerDied","Data":"fef570eb96743881f6c50086e10a9ad34baf666db5432c79760e2ba6b2b3476b"}
Jul 28 19:50:55.930503 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:55.930062    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp" event={"ID":"9ea6686d-af93-4f4d-9c5d-7472ea97563f","Type":"ContainerStarted","Data":"d1091720163c5d072edcb7f11919d7d1e6b264c90db322b0b2f9885be09e4889"}
Jul 28 19:50:57.845385 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.845340    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"]
Jul 28 19:50:57.858069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.857967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:57.858699 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.858670    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"]
Jul 28 19:50:57.860601 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.860578    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 28 19:50:57.861862 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.861836    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x6r8q\""
Jul 28 19:50:57.861862 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.861846    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 28 19:50:57.891484 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.891449    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bb1318-fe7d-4dcc-b59c-9cb4648d3919-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dw7xc\" (UID: \"54bb1318-fe7d-4dcc-b59c-9cb4648d3919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:57.891660 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.891545    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5b94\" (UniqueName: \"kubernetes.io/projected/54bb1318-fe7d-4dcc-b59c-9cb4648d3919-kube-api-access-j5b94\") pod \"cert-manager-cainjector-559cdd8c87-dw7xc\" (UID: \"54bb1318-fe7d-4dcc-b59c-9cb4648d3919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:57.992665 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.992628    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5b94\" (UniqueName: \"kubernetes.io/projected/54bb1318-fe7d-4dcc-b59c-9cb4648d3919-kube-api-access-j5b94\") pod \"cert-manager-cainjector-559cdd8c87-dw7xc\" (UID: \"54bb1318-fe7d-4dcc-b59c-9cb4648d3919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:57.992875 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:57.992694    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bb1318-fe7d-4dcc-b59c-9cb4648d3919-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dw7xc\" (UID: \"54bb1318-fe7d-4dcc-b59c-9cb4648d3919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:58.001036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.001004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5b94\" (UniqueName: \"kubernetes.io/projected/54bb1318-fe7d-4dcc-b59c-9cb4648d3919-kube-api-access-j5b94\") pod \"cert-manager-cainjector-559cdd8c87-dw7xc\" (UID: \"54bb1318-fe7d-4dcc-b59c-9cb4648d3919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:58.001382 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.001362    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/54bb1318-fe7d-4dcc-b59c-9cb4648d3919-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-dw7xc\" (UID: \"54bb1318-fe7d-4dcc-b59c-9cb4648d3919\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:58.180057 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.180023    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"
Jul 28 19:50:58.311423 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.311397    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc"]
Jul 28 19:50:58.313659 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:50:58.313629    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54bb1318_fe7d_4dcc_b59c_9cb4648d3919.slice/crio-b96afff776f946df6ddb41a92794d34698509e012ed40f5faad6caaaf93ab7d8 WatchSource:0}: Error finding container b96afff776f946df6ddb41a92794d34698509e012ed40f5faad6caaaf93ab7d8: Status 404 returned error can't find the container with id b96afff776f946df6ddb41a92794d34698509e012ed40f5faad6caaaf93ab7d8
Jul 28 19:50:58.943735 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.943694    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerID="f81c928d95e2e2563af871f3c9b151d68cd68c9b25da73f5fec126abbee9ec17" exitCode=0
Jul 28 19:50:58.944286 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.943788    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp" event={"ID":"9ea6686d-af93-4f4d-9c5d-7472ea97563f","Type":"ContainerDied","Data":"f81c928d95e2e2563af871f3c9b151d68cd68c9b25da73f5fec126abbee9ec17"}
Jul 28 19:50:58.945086 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:58.945065    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc" event={"ID":"54bb1318-fe7d-4dcc-b59c-9cb4648d3919","Type":"ContainerStarted","Data":"b96afff776f946df6ddb41a92794d34698509e012ed40f5faad6caaaf93ab7d8"}
Jul 28 19:50:59.952399 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:59.952358    2578 generic.go:356] "Generic (PLEG): container finished" podID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerID="ac7c6148d3da8381366bb254ace6f08c44470efd64b1aa628e0ab55a29705ab9" exitCode=0
Jul 28 19:50:59.952860 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:50:59.952445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp" event={"ID":"9ea6686d-af93-4f4d-9c5d-7472ea97563f","Type":"ContainerDied","Data":"ac7c6148d3da8381366bb254ace6f08c44470efd64b1aa628e0ab55a29705ab9"}
Jul 28 19:51:01.095056 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.095027    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:51:01.117074 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.117038    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72b8r\" (UniqueName: \"kubernetes.io/projected/9ea6686d-af93-4f4d-9c5d-7472ea97563f-kube-api-access-72b8r\") pod \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") "
Jul 28 19:51:01.117219 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.117097    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-util\") pod \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") "
Jul 28 19:51:01.117219 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.117205    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-bundle\") pod \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\" (UID: \"9ea6686d-af93-4f4d-9c5d-7472ea97563f\") "
Jul 28 19:51:01.117585 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.117557    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-bundle" (OuterVolumeSpecName: "bundle") pod "9ea6686d-af93-4f4d-9c5d-7472ea97563f" (UID: "9ea6686d-af93-4f4d-9c5d-7472ea97563f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:51:01.119309 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.119285    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea6686d-af93-4f4d-9c5d-7472ea97563f-kube-api-access-72b8r" (OuterVolumeSpecName: "kube-api-access-72b8r") pod "9ea6686d-af93-4f4d-9c5d-7472ea97563f" (UID: "9ea6686d-af93-4f4d-9c5d-7472ea97563f"). InnerVolumeSpecName "kube-api-access-72b8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:51:01.124218 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.124190    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-util" (OuterVolumeSpecName: "util") pod "9ea6686d-af93-4f4d-9c5d-7472ea97563f" (UID: "9ea6686d-af93-4f4d-9c5d-7472ea97563f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:51:01.217800 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.217723    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72b8r\" (UniqueName: \"kubernetes.io/projected/9ea6686d-af93-4f4d-9c5d-7472ea97563f-kube-api-access-72b8r\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:51:01.217800 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.217756    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:51:01.217800 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.217766    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9ea6686d-af93-4f4d-9c5d-7472ea97563f-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:51:01.962842 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.962812    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp"
Jul 28 19:51:01.963022 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.962811    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fhhmsp" event={"ID":"9ea6686d-af93-4f4d-9c5d-7472ea97563f","Type":"ContainerDied","Data":"d1091720163c5d072edcb7f11919d7d1e6b264c90db322b0b2f9885be09e4889"}
Jul 28 19:51:01.963022 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.962939    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1091720163c5d072edcb7f11919d7d1e6b264c90db322b0b2f9885be09e4889"
Jul 28 19:51:01.964473 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.964441    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc" event={"ID":"54bb1318-fe7d-4dcc-b59c-9cb4648d3919","Type":"ContainerStarted","Data":"16221ce732a75c84c72ec2658b275ab9235445509c4f6e20e3b0dcc6cd543ed7"}
Jul 28 19:51:01.980974 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:01.980918    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-dw7xc" podStartSLOduration=1.8829232 podStartE2EDuration="4.980901972s" totalImagesPullingTime="3.097978772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:50:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:51:01.97941555 +0000 UTC m=+264.879632393" watchObservedRunningTime="2026-07-28 19:51:01.980901972 +0000 UTC m=+264.881118814"
Jul 28 19:51:12.444888 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.444847    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"]
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445160    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="util"
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445174    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="util"
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445190    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="pull"
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445195    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="pull"
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445205    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="extract"
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445210    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="extract"
Jul 28 19:51:12.445403 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.445257    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea6686d-af93-4f4d-9c5d-7472ea97563f" containerName="extract"
Jul 28 19:51:12.495293 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.495222    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"]
Jul 28 19:51:12.495517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.495350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.498161 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.498133    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 19:51:12.499256 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.499240    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 19:51:12.499366 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.499299    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5m7hq\""
Jul 28 19:51:12.609458 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.609417    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1e18ad1-0a6a-482b-934f-ba587e980a1f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jsn5w\" (UID: \"e1e18ad1-0a6a-482b-934f-ba587e980a1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.609709 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.609470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5jsn\" (UniqueName: \"kubernetes.io/projected/e1e18ad1-0a6a-482b-934f-ba587e980a1f-kube-api-access-l5jsn\") pod \"openshift-lws-operator-fd8ccff4c-jsn5w\" (UID: \"e1e18ad1-0a6a-482b-934f-ba587e980a1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.710180 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.710088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5jsn\" (UniqueName: \"kubernetes.io/projected/e1e18ad1-0a6a-482b-934f-ba587e980a1f-kube-api-access-l5jsn\") pod \"openshift-lws-operator-fd8ccff4c-jsn5w\" (UID: \"e1e18ad1-0a6a-482b-934f-ba587e980a1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.710180 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.710165    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1e18ad1-0a6a-482b-934f-ba587e980a1f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jsn5w\" (UID: \"e1e18ad1-0a6a-482b-934f-ba587e980a1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.710564 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.710545    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1e18ad1-0a6a-482b-934f-ba587e980a1f-tmp\") pod \"openshift-lws-operator-fd8ccff4c-jsn5w\" (UID: \"e1e18ad1-0a6a-482b-934f-ba587e980a1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.717995 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.717974    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5jsn\" (UniqueName: \"kubernetes.io/projected/e1e18ad1-0a6a-482b-934f-ba587e980a1f-kube-api-access-l5jsn\") pod \"openshift-lws-operator-fd8ccff4c-jsn5w\" (UID: \"e1e18ad1-0a6a-482b-934f-ba587e980a1f\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.804949 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.804906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"
Jul 28 19:51:12.960354 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:12.960291    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w"]
Jul 28 19:51:12.963147 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:51:12.963119    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1e18ad1_0a6a_482b_934f_ba587e980a1f.slice/crio-a3ce1ddad90789252a90c8f35dbc7851e02b6ced93ef1de0e912b4625e69f941 WatchSource:0}: Error finding container a3ce1ddad90789252a90c8f35dbc7851e02b6ced93ef1de0e912b4625e69f941: Status 404 returned error can't find the container with id a3ce1ddad90789252a90c8f35dbc7851e02b6ced93ef1de0e912b4625e69f941
Jul 28 19:51:13.012891 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:13.012852    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w" event={"ID":"e1e18ad1-0a6a-482b-934f-ba587e980a1f","Type":"ContainerStarted","Data":"a3ce1ddad90789252a90c8f35dbc7851e02b6ced93ef1de0e912b4625e69f941"}
Jul 28 19:51:17.031824 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:17.031785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w" event={"ID":"e1e18ad1-0a6a-482b-934f-ba587e980a1f","Type":"ContainerStarted","Data":"e6d6d64a310abaafcf2f53048a730f0afeb5b6353bfff33da9777fc26b8423ea"}
Jul 28 19:51:17.047819 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:17.047762    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-jsn5w" podStartSLOduration=1.5026046549999998 podStartE2EDuration="5.047747131s" totalImagesPullingTime="3.545142476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:51:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:51:17.046785001 +0000 UTC m=+279.947001841" watchObservedRunningTime="2026-07-28 19:51:17.047747131 +0000 UTC m=+279.947963973"
Jul 28 19:51:19.892728 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.892685    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"]
Jul 28 19:51:19.895020 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.895002    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:19.897551 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.897528    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 28 19:51:19.897692 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.897596    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 28 19:51:19.897754 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.897695    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ssthg\""
Jul 28 19:51:19.903445 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.903421    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"]
Jul 28 19:51:19.971780 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.971736    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:19.971954 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.971831    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjsl\" (UniqueName: \"kubernetes.io/projected/d6550c1a-9188-4a3c-af79-40d05d6ff0de-kube-api-access-7wjsl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:19.971954 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:19.971885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.072761 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.072721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.072761 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.072771    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjsl\" (UniqueName: \"kubernetes.io/projected/d6550c1a-9188-4a3c-af79-40d05d6ff0de-kube-api-access-7wjsl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.072999 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.072796    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.073124 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.073102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.073186 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.073165    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.081382 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.081345    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjsl\" (UniqueName: \"kubernetes.io/projected/d6550c1a-9188-4a3c-af79-40d05d6ff0de-kube-api-access-7wjsl\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.205382 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.205296    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:20.354023 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:20.353997    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"]
Jul 28 19:51:20.356629 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:51:20.356601    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6550c1a_9188_4a3c_af79_40d05d6ff0de.slice/crio-05da0faefa48303ca5edea0a96ae3b90156f600818b1340787e744aaa0d5414b WatchSource:0}: Error finding container 05da0faefa48303ca5edea0a96ae3b90156f600818b1340787e744aaa0d5414b: Status 404 returned error can't find the container with id 05da0faefa48303ca5edea0a96ae3b90156f600818b1340787e744aaa0d5414b
Jul 28 19:51:21.053434 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:21.053395    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerID="39d7dd4e08f415abcddb287e7571be2b8862a8371e989d83751fdc46eb77a1e6" exitCode=0
Jul 28 19:51:21.053881 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:21.053482    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs" event={"ID":"d6550c1a-9188-4a3c-af79-40d05d6ff0de","Type":"ContainerDied","Data":"39d7dd4e08f415abcddb287e7571be2b8862a8371e989d83751fdc46eb77a1e6"}
Jul 28 19:51:21.053881 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:21.053532    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs" event={"ID":"d6550c1a-9188-4a3c-af79-40d05d6ff0de","Type":"ContainerStarted","Data":"05da0faefa48303ca5edea0a96ae3b90156f600818b1340787e744aaa0d5414b"}
Jul 28 19:51:22.059135 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:22.059101    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerID="bdb0e1d94115fbe0934b619771a707a6c6e546cb2c20522353229800de2f23c7" exitCode=0
Jul 28 19:51:22.059533 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:22.059185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs" event={"ID":"d6550c1a-9188-4a3c-af79-40d05d6ff0de","Type":"ContainerDied","Data":"bdb0e1d94115fbe0934b619771a707a6c6e546cb2c20522353229800de2f23c7"}
Jul 28 19:51:23.065394 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:23.065360    2578 generic.go:356] "Generic (PLEG): container finished" podID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerID="cd5efec10a9ac81f12bb4ed15bf10ba8f4db5677219b4f379d4cfb9659a3b920" exitCode=0
Jul 28 19:51:23.065791 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:23.065445    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs" event={"ID":"d6550c1a-9188-4a3c-af79-40d05d6ff0de","Type":"ContainerDied","Data":"cd5efec10a9ac81f12bb4ed15bf10ba8f4db5677219b4f379d4cfb9659a3b920"}
Jul 28 19:51:24.200710 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.200679    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:24.313662 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.313614    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-util\") pod \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") "
Jul 28 19:51:24.313847 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.313677    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-bundle\") pod \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") "
Jul 28 19:51:24.313847 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.313748    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7wjsl\" (UniqueName: \"kubernetes.io/projected/d6550c1a-9188-4a3c-af79-40d05d6ff0de-kube-api-access-7wjsl\") pod \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\" (UID: \"d6550c1a-9188-4a3c-af79-40d05d6ff0de\") "
Jul 28 19:51:24.314448 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.314421    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-bundle" (OuterVolumeSpecName: "bundle") pod "d6550c1a-9188-4a3c-af79-40d05d6ff0de" (UID: "d6550c1a-9188-4a3c-af79-40d05d6ff0de"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:51:24.316006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.315977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6550c1a-9188-4a3c-af79-40d05d6ff0de-kube-api-access-7wjsl" (OuterVolumeSpecName: "kube-api-access-7wjsl") pod "d6550c1a-9188-4a3c-af79-40d05d6ff0de" (UID: "d6550c1a-9188-4a3c-af79-40d05d6ff0de"). InnerVolumeSpecName "kube-api-access-7wjsl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:51:24.319328 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.319306    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-util" (OuterVolumeSpecName: "util") pod "d6550c1a-9188-4a3c-af79-40d05d6ff0de" (UID: "d6550c1a-9188-4a3c-af79-40d05d6ff0de"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:51:24.415232 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.415191    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjsl\" (UniqueName: \"kubernetes.io/projected/d6550c1a-9188-4a3c-af79-40d05d6ff0de-kube-api-access-7wjsl\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:51:24.415232 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.415224    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:51:24.415232 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:24.415236    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d6550c1a-9188-4a3c-af79-40d05d6ff0de-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:51:25.076037 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:25.075998    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs" event={"ID":"d6550c1a-9188-4a3c-af79-40d05d6ff0de","Type":"ContainerDied","Data":"05da0faefa48303ca5edea0a96ae3b90156f600818b1340787e744aaa0d5414b"}
Jul 28 19:51:25.076037 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:25.076036    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af98sjs"
Jul 28 19:51:25.076037 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:25.076046    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05da0faefa48303ca5edea0a96ae3b90156f600818b1340787e744aaa0d5414b"
Jul 28 19:51:28.085316 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085261    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"]
Jul 28 19:51:28.085744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085666    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="pull"
Jul 28 19:51:28.085744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085681    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="pull"
Jul 28 19:51:28.085744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085705    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="util"
Jul 28 19:51:28.085744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085711    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="util"
Jul 28 19:51:28.085744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085724    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="extract"
Jul 28 19:51:28.085744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085730    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="extract"
Jul 28 19:51:28.085967 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.085783    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6550c1a-9188-4a3c-af79-40d05d6ff0de" containerName="extract"
Jul 28 19:51:28.088435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.088413    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.091056 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.091036    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 19:51:28.092285 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.092240    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-24knk\""
Jul 28 19:51:28.092448 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.092261    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 19:51:28.092448 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.092305    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 19:51:28.098070 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.098044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"]
Jul 28 19:51:28.150567 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.150523    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-manager-config\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.150567 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.150570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-cert\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.150805 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.150608    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-metrics-cert\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.150805 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.150637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mc8g\" (UniqueName: \"kubernetes.io/projected/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-kube-api-access-6mc8g\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.251539 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.251498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-manager-config\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.251539 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.251546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-cert\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.251882 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.251579    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-metrics-cert\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.251882 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.251609    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mc8g\" (UniqueName: \"kubernetes.io/projected/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-kube-api-access-6mc8g\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.252984 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.252951    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-manager-config\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.254395 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.254365    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-cert\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.254395 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.254384    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-metrics-cert\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.261944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.261920    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mc8g\" (UniqueName: \"kubernetes.io/projected/3b2e7cdc-aeb3-421e-a530-5fec50e87c8b-kube-api-access-6mc8g\") pod \"lws-controller-manager-6846c7dcc9-fhp67\" (UID: \"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b\") " pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.398464 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.398374    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:28.536508 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:28.536481    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"]
Jul 28 19:51:28.538539 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:51:28.538515    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b2e7cdc_aeb3_421e_a530_5fec50e87c8b.slice/crio-6492de9c3d4f76e6d412b56f6b4b0bd7cba872c113f605ee1b48d56f5fe8cdee WatchSource:0}: Error finding container 6492de9c3d4f76e6d412b56f6b4b0bd7cba872c113f605ee1b48d56f5fe8cdee: Status 404 returned error can't find the container with id 6492de9c3d4f76e6d412b56f6b4b0bd7cba872c113f605ee1b48d56f5fe8cdee
Jul 28 19:51:29.094967 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:29.094930    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67" event={"ID":"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b","Type":"ContainerStarted","Data":"6492de9c3d4f76e6d412b56f6b4b0bd7cba872c113f605ee1b48d56f5fe8cdee"}
Jul 28 19:51:31.106494 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:31.106455    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67" event={"ID":"3b2e7cdc-aeb3-421e-a530-5fec50e87c8b","Type":"ContainerStarted","Data":"46fd1c7fd519a30d5794aa09f714044bea05fb23eb85a19512faddb6ccc6a8d9"}
Jul 28 19:51:31.106872 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:31.106538    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:31.124753 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:31.124704    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67" podStartSLOduration=1.3192341220000001 podStartE2EDuration="3.124688857s" totalImagesPullingTime="1.805454735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:51:31.123603242 +0000 UTC m=+294.023820086" watchObservedRunningTime="2026-07-28 19:51:31.124688857 +0000 UTC m=+294.024905720"
Jul 28 19:51:32.611246 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.611209    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"]
Jul 28 19:51:32.614864 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.614844    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.619090 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.619061    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 19:51:32.619090 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.619066    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f6pc8\""
Jul 28 19:51:32.619301 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.619065    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 19:51:32.619435 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.619419    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 19:51:32.619498 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.619418    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 19:51:32.656568 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.656538    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"]
Jul 28 19:51:32.690196 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.690161    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78513af9-1638-4bc7-981b-1a2219c1160c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.690384 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.690211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7f6\" (UniqueName: \"kubernetes.io/projected/78513af9-1638-4bc7-981b-1a2219c1160c-kube-api-access-sz7f6\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.690384 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.690317    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78513af9-1638-4bc7-981b-1a2219c1160c-manifests\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.690384 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.690349    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78513af9-1638-4bc7-981b-1a2219c1160c-tmp\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.690384 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.690371    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78513af9-1638-4bc7-981b-1a2219c1160c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.791492 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.791447    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7f6\" (UniqueName: \"kubernetes.io/projected/78513af9-1638-4bc7-981b-1a2219c1160c-kube-api-access-sz7f6\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.791640 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.791504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78513af9-1638-4bc7-981b-1a2219c1160c-manifests\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.791640 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.791525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78513af9-1638-4bc7-981b-1a2219c1160c-tmp\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.791640 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.791546    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78513af9-1638-4bc7-981b-1a2219c1160c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.791640 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.791575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78513af9-1638-4bc7-981b-1a2219c1160c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.792031 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.792011    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/78513af9-1638-4bc7-981b-1a2219c1160c-manifests\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.794170 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.794146    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/78513af9-1638-4bc7-981b-1a2219c1160c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.794306 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.794176    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/78513af9-1638-4bc7-981b-1a2219c1160c-tmp\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.794364 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.794324    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/78513af9-1638-4bc7-981b-1a2219c1160c-webhook-cert\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.800309 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.800284    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7f6\" (UniqueName: \"kubernetes.io/projected/78513af9-1638-4bc7-981b-1a2219c1160c-kube-api-access-sz7f6\") pod \"opendatahub-operator-controller-manager-85967bc8b9-cwvcg\" (UID: \"78513af9-1638-4bc7-981b-1a2219c1160c\") " pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:32.925649 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:32.925559    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:33.065117 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:33.064981    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"]
Jul 28 19:51:33.067769 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:51:33.067737    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78513af9_1638_4bc7_981b_1a2219c1160c.slice/crio-1e414030e86f4ead6ebd390fc83918678631a9220248d442103c9f1943ddb64a WatchSource:0}: Error finding container 1e414030e86f4ead6ebd390fc83918678631a9220248d442103c9f1943ddb64a: Status 404 returned error can't find the container with id 1e414030e86f4ead6ebd390fc83918678631a9220248d442103c9f1943ddb64a
Jul 28 19:51:33.117244 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:33.117193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg" event={"ID":"78513af9-1638-4bc7-981b-1a2219c1160c","Type":"ContainerStarted","Data":"1e414030e86f4ead6ebd390fc83918678631a9220248d442103c9f1943ddb64a"}
Jul 28 19:51:37.140469 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:37.140423    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg" event={"ID":"78513af9-1638-4bc7-981b-1a2219c1160c","Type":"ContainerStarted","Data":"21e05a3205b6400fedcd0f36ba9a4dda72dd6d673cd597bdb07488a966f3b281"}
Jul 28 19:51:37.485751 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:37.485724    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:51:37.486901 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:37.486877    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:51:37.498024 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:37.498000    2578 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 19:51:38.147033 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:38.146999    2578 generic.go:356] "Generic (PLEG): container finished" podID="78513af9-1638-4bc7-981b-1a2219c1160c" containerID="21e05a3205b6400fedcd0f36ba9a4dda72dd6d673cd597bdb07488a966f3b281" exitCode=0
Jul 28 19:51:38.148562 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:38.147083    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg" event={"ID":"78513af9-1638-4bc7-981b-1a2219c1160c","Type":"ContainerDied","Data":"21e05a3205b6400fedcd0f36ba9a4dda72dd6d673cd597bdb07488a966f3b281"}
Jul 28 19:51:39.153621 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:39.153582    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg" event={"ID":"78513af9-1638-4bc7-981b-1a2219c1160c","Type":"ContainerStarted","Data":"f09f81b31cd7b270bbc3fe2d5f4c30e060d400ab25de6ace7320f4367c203cc2"}
Jul 28 19:51:39.154013 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:39.153687    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:39.178893 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:39.178837    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg" podStartSLOduration=2.635385313 podStartE2EDuration="7.178819499s" totalImagesPullingTime="3.455121s" totalInitContainerRuntime="1.088313186s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:51:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:51:39.176114055 +0000 UTC m=+302.076330901" watchObservedRunningTime="2026-07-28 19:51:39.178819499 +0000 UTC m=+302.079036344"
Jul 28 19:51:42.113580 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:42.113548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6846c7dcc9-fhp67"
Jul 28 19:51:50.161111 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:50.161079    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-85967bc8b9-cwvcg"
Jul 28 19:51:59.919862 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.919824    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"]
Jul 28 19:51:59.922102 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.922084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:51:59.924687 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.924660    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 28 19:51:59.924860 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.924741    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 19:51:59.924860 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.924771    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 28 19:51:59.924860 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.924802    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fnbcf\""
Jul 28 19:51:59.924860 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.924742    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 28 19:51:59.936715 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:51:59.936689    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"]
Jul 28 19:52:00.024723 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024679    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.024723 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.024934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024771    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f8910-418b-457e-8e24-18f4976a18bd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.024934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024816    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/836f8910-418b-457e-8e24-18f4976a18bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.024934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024845    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b768k\" (UniqueName: \"kubernetes.io/projected/836f8910-418b-457e-8e24-18f4976a18bd-kube-api-access-b768k\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.024934 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024866    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f8910-418b-457e-8e24-18f4976a18bd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.025069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.024932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126226 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/836f8910-418b-457e-8e24-18f4976a18bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126226 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126229    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b768k\" (UniqueName: \"kubernetes.io/projected/836f8910-418b-457e-8e24-18f4976a18bd-kube-api-access-b768k\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f8910-418b-457e-8e24-18f4976a18bd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126351    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126414    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.126517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126446    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f8910-418b-457e-8e24-18f4976a18bd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.127025 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.126999    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/836f8910-418b-457e-8e24-18f4976a18bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.129019 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.128993    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/836f8910-418b-457e-8e24-18f4976a18bd-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.129137 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.129079    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.129385 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.129367    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.129467 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.129452    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/836f8910-418b-457e-8e24-18f4976a18bd-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.134401 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.134374    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b768k\" (UniqueName: \"kubernetes.io/projected/836f8910-418b-457e-8e24-18f4976a18bd-kube-api-access-b768k\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.134669 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.134654    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/836f8910-418b-457e-8e24-18f4976a18bd-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jkqdq\" (UID: \"836f8910-418b-457e-8e24-18f4976a18bd\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.231339 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.231229    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:00.389119 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.388921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"]
Jul 28 19:52:00.392095 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:52:00.392063    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod836f8910_418b_457e_8e24_18f4976a18bd.slice/crio-7c257d959c10f537f44099c0173615204a3f0d2ca127033ec36cd024aa918993 WatchSource:0}: Error finding container 7c257d959c10f537f44099c0173615204a3f0d2ca127033ec36cd024aa918993: Status 404 returned error can't find the container with id 7c257d959c10f537f44099c0173615204a3f0d2ca127033ec36cd024aa918993
Jul 28 19:52:00.394001 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:00.393983    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 19:52:01.263106 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:01.263069    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq" event={"ID":"836f8910-418b-457e-8e24-18f4976a18bd","Type":"ContainerStarted","Data":"7c257d959c10f537f44099c0173615204a3f0d2ca127033ec36cd024aa918993"}
Jul 28 19:52:03.574734 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:03.574695    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 19:52:03.575017 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:03.574774    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 19:52:04.280259 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:04.280211    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq" event={"ID":"836f8910-418b-457e-8e24-18f4976a18bd","Type":"ContainerStarted","Data":"28a3b1dac7cd6f99cd226e98ed59e1aeab4060497173b46e74733978576dcf87"}
Jul 28 19:52:04.280491 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:04.280363    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:52:04.281890 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:04.281862    2578 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-jkqdq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 19:52:04.282006 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:04.281921    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq" podUID="836f8910-418b-457e-8e24-18f4976a18bd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 19:52:04.320193 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:04.320129    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq" podStartSLOduration=2.139781482 podStartE2EDuration="5.320108757s" totalImagesPullingTime="3.180327275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:52:04.318796637 +0000 UTC m=+327.219013480" watchObservedRunningTime="2026-07-28 19:52:04.320108757 +0000 UTC m=+327.220325601"
Jul 28 19:52:05.285749 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:52:05.285720    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jkqdq"
Jul 28 19:53:09.068154 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.068115    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"]
Jul 28 19:53:09.071335 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.071315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.073782 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.073759    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mflq8\""
Jul 28 19:53:09.073930 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.073809    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 19:53:09.074822 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.074806    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 19:53:09.079461 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.079441    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"]
Jul 28 19:53:09.217872 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.217833    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgqv8\" (UniqueName: \"kubernetes.io/projected/24f3f95e-89a0-4603-9082-e96fc7f35112-kube-api-access-bgqv8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.218048 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.217907    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.218048 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.217945    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.318650 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.318553    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.318650 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.318612    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.318859 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.318654    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgqv8\" (UniqueName: \"kubernetes.io/projected/24f3f95e-89a0-4603-9082-e96fc7f35112-kube-api-access-bgqv8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.319046 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.319019    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.319095 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.319080    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.327104 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.327078    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgqv8\" (UniqueName: \"kubernetes.io/projected/24f3f95e-89a0-4603-9082-e96fc7f35112-kube-api-access-bgqv8\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.382106 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.382059    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:09.520379 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.520347    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"]
Jul 28 19:53:09.522239 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:53:09.522209    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f3f95e_89a0_4603_9082_e96fc7f35112.slice/crio-5bf1742fc1a163e4c255fd04b19a40e9af3eea438c7ceeb90783c68fc12709a2 WatchSource:0}: Error finding container 5bf1742fc1a163e4c255fd04b19a40e9af3eea438c7ceeb90783c68fc12709a2: Status 404 returned error can't find the container with id 5bf1742fc1a163e4c255fd04b19a40e9af3eea438c7ceeb90783c68fc12709a2
Jul 28 19:53:09.580960 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.580924    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd" event={"ID":"24f3f95e-89a0-4603-9082-e96fc7f35112","Type":"ContainerStarted","Data":"5bf1742fc1a163e4c255fd04b19a40e9af3eea438c7ceeb90783c68fc12709a2"}
Jul 28 19:53:09.868790 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.868687    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"]
Jul 28 19:53:09.872233 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.872214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:09.880672 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:09.880642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"]
Jul 28 19:53:10.026593 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.026541    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.026593 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.026598    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.026828 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.026715    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2ts2\" (UniqueName: \"kubernetes.io/projected/46885940-23b6-4079-8abf-92ce1a09bde1-kube-api-access-r2ts2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.127943 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.127858    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2ts2\" (UniqueName: \"kubernetes.io/projected/46885940-23b6-4079-8abf-92ce1a09bde1-kube-api-access-r2ts2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.127943 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.127920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.127943 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.127943    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.128430 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.128300    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.128430 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.128341    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.138503 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.138472    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2ts2\" (UniqueName: \"kubernetes.io/projected/46885940-23b6-4079-8abf-92ce1a09bde1-kube-api-access-r2ts2\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.182286 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.182233    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:10.313250 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.313219    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"]
Jul 28 19:53:10.314922 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:53:10.314894    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46885940_23b6_4079_8abf_92ce1a09bde1.slice/crio-3bb499ea348f1cdb96fa313cc2a0eebd542899c361672468fa2e64be29a1a3bf WatchSource:0}: Error finding container 3bb499ea348f1cdb96fa313cc2a0eebd542899c361672468fa2e64be29a1a3bf: Status 404 returned error can't find the container with id 3bb499ea348f1cdb96fa313cc2a0eebd542899c361672468fa2e64be29a1a3bf
Jul 28 19:53:10.464854 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.464818    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"]
Jul 28 19:53:10.468546 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.468525    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.476139 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.476114    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"]
Jul 28 19:53:10.531064 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.531026    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.531201 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.531087    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6zjh\" (UniqueName: \"kubernetes.io/projected/5aec8309-bef8-4833-918b-fc5d8f7f14d6-kube-api-access-f6zjh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.531201 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.531147    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.586605 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.586569    2578 generic.go:356] "Generic (PLEG): container finished" podID="46885940-23b6-4079-8abf-92ce1a09bde1" containerID="38a56632a59af1ca90b5760bacaf48c449afe0b0231cd1e62c9332ba8d041328" exitCode=0
Jul 28 19:53:10.586750 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.586658    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq" event={"ID":"46885940-23b6-4079-8abf-92ce1a09bde1","Type":"ContainerDied","Data":"38a56632a59af1ca90b5760bacaf48c449afe0b0231cd1e62c9332ba8d041328"}
Jul 28 19:53:10.586750 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.586714    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq" event={"ID":"46885940-23b6-4079-8abf-92ce1a09bde1","Type":"ContainerStarted","Data":"3bb499ea348f1cdb96fa313cc2a0eebd542899c361672468fa2e64be29a1a3bf"}
Jul 28 19:53:10.588336 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.588313    2578 generic.go:356] "Generic (PLEG): container finished" podID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerID="9af24f6e7ad9f6c2fa2bfbb31f60ef93004d052e5ee9e5964aec82f542bb84ef" exitCode=0
Jul 28 19:53:10.588419 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.588396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd" event={"ID":"24f3f95e-89a0-4603-9082-e96fc7f35112","Type":"ContainerDied","Data":"9af24f6e7ad9f6c2fa2bfbb31f60ef93004d052e5ee9e5964aec82f542bb84ef"}
Jul 28 19:53:10.631924 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.631893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.632117 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.631942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6zjh\" (UniqueName: \"kubernetes.io/projected/5aec8309-bef8-4833-918b-fc5d8f7f14d6-kube-api-access-f6zjh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.632117 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.631982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.632348 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.632328    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.632414 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.632330    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.641039 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.640978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6zjh\" (UniqueName: \"kubernetes.io/projected/5aec8309-bef8-4833-918b-fc5d8f7f14d6-kube-api-access-f6zjh\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.793685 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.793642    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:10.930373 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:10.930288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"]
Jul 28 19:53:10.933481 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:53:10.933457    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aec8309_bef8_4833_918b_fc5d8f7f14d6.slice/crio-f0da3d08d9afc920e61069ff6a4d29744346511d17d2be157c16cd00f0f767fe WatchSource:0}: Error finding container f0da3d08d9afc920e61069ff6a4d29744346511d17d2be157c16cd00f0f767fe: Status 404 returned error can't find the container with id f0da3d08d9afc920e61069ff6a4d29744346511d17d2be157c16cd00f0f767fe
Jul 28 19:53:11.071150 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.071117    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"]
Jul 28 19:53:11.074416 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.074397    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.085761 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.085731    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"]
Jul 28 19:53:11.136990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.136951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.136990 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.136989    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfsmf\" (UniqueName: \"kubernetes.io/projected/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-kube-api-access-zfsmf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.137474 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.137037    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.238004 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.237898    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.238004 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.237940    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfsmf\" (UniqueName: \"kubernetes.io/projected/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-kube-api-access-zfsmf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.238004 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.237986    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.238422 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.238401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.238496 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.238401    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.247545 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.247519    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfsmf\" (UniqueName: \"kubernetes.io/projected/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-kube-api-access-zfsmf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.390413 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.390369    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:11.560799 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.560774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"]
Jul 28 19:53:11.569144 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:53:11.569116    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd04bb88d_00c0_46fd_bc1d_0138d0ea3f91.slice/crio-c001b0e60782333229dc1a07a3abb34a1a65de79e6f846c44def7fe99ad5a5c5 WatchSource:0}: Error finding container c001b0e60782333229dc1a07a3abb34a1a65de79e6f846c44def7fe99ad5a5c5: Status 404 returned error can't find the container with id c001b0e60782333229dc1a07a3abb34a1a65de79e6f846c44def7fe99ad5a5c5
Jul 28 19:53:11.595022 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.594991    2578 generic.go:356] "Generic (PLEG): container finished" podID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerID="e27a7d32b46418abd8996a31fd9d6ca68f468ae47157061033390bdd331edded" exitCode=0
Jul 28 19:53:11.595141 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.595027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm" event={"ID":"5aec8309-bef8-4833-918b-fc5d8f7f14d6","Type":"ContainerDied","Data":"e27a7d32b46418abd8996a31fd9d6ca68f468ae47157061033390bdd331edded"}
Jul 28 19:53:11.595141 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.595071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm" event={"ID":"5aec8309-bef8-4833-918b-fc5d8f7f14d6","Type":"ContainerStarted","Data":"f0da3d08d9afc920e61069ff6a4d29744346511d17d2be157c16cd00f0f767fe"}
Jul 28 19:53:11.596215 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.596193    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq" event={"ID":"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91","Type":"ContainerStarted","Data":"c001b0e60782333229dc1a07a3abb34a1a65de79e6f846c44def7fe99ad5a5c5"}
Jul 28 19:53:11.598089 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.598070    2578 generic.go:356] "Generic (PLEG): container finished" podID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerID="c6d2966f2d837c12dafd37e9541884aa0f9d8e16900a3b7ab3bba08584b2cdaa" exitCode=0
Jul 28 19:53:11.598171 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:11.598150    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd" event={"ID":"24f3f95e-89a0-4603-9082-e96fc7f35112","Type":"ContainerDied","Data":"c6d2966f2d837c12dafd37e9541884aa0f9d8e16900a3b7ab3bba08584b2cdaa"}
Jul 28 19:53:12.605085 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.604994    2578 generic.go:356] "Generic (PLEG): container finished" podID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerID="46d275220a08231858bf7f551e92650953733b18180f4a0df151052c6054e807" exitCode=0
Jul 28 19:53:12.605511 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.605082    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm" event={"ID":"5aec8309-bef8-4833-918b-fc5d8f7f14d6","Type":"ContainerDied","Data":"46d275220a08231858bf7f551e92650953733b18180f4a0df151052c6054e807"}
Jul 28 19:53:12.607012 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.606988    2578 generic.go:356] "Generic (PLEG): container finished" podID="46885940-23b6-4079-8abf-92ce1a09bde1" containerID="6500f45accffe82b9e5b2396c67155a37456ca62cc164cf9eec4add0e77c3bb8" exitCode=0
Jul 28 19:53:12.607092 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.607058    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq" event={"ID":"46885940-23b6-4079-8abf-92ce1a09bde1","Type":"ContainerDied","Data":"6500f45accffe82b9e5b2396c67155a37456ca62cc164cf9eec4add0e77c3bb8"}
Jul 28 19:53:12.608624 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.608599    2578 generic.go:356] "Generic (PLEG): container finished" podID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerID="c16236b846809da920d4c302363d53a609c45416c2a6ec66e29d3f0da7833fae" exitCode=0
Jul 28 19:53:12.608722 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.608624    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq" event={"ID":"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91","Type":"ContainerDied","Data":"c16236b846809da920d4c302363d53a609c45416c2a6ec66e29d3f0da7833fae"}
Jul 28 19:53:12.610944 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.610922    2578 generic.go:356] "Generic (PLEG): container finished" podID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerID="18b7ad51384cd83f1a0c55347e71a7d182bc7980d46a9519b9ea12f381a69e02" exitCode=0
Jul 28 19:53:12.611031 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:12.610992    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd" event={"ID":"24f3f95e-89a0-4603-9082-e96fc7f35112","Type":"ContainerDied","Data":"18b7ad51384cd83f1a0c55347e71a7d182bc7980d46a9519b9ea12f381a69e02"}
Jul 28 19:53:13.619284 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.619188    2578 generic.go:356] "Generic (PLEG): container finished" podID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerID="8416645fa2bb125d01db0fb6aa8479b50b6952d54b580aaa5d895f00197b6125" exitCode=0
Jul 28 19:53:13.619713 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.619298    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm" event={"ID":"5aec8309-bef8-4833-918b-fc5d8f7f14d6","Type":"ContainerDied","Data":"8416645fa2bb125d01db0fb6aa8479b50b6952d54b580aaa5d895f00197b6125"}
Jul 28 19:53:13.621421 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.621393    2578 generic.go:356] "Generic (PLEG): container finished" podID="46885940-23b6-4079-8abf-92ce1a09bde1" containerID="513ff881fd3d0dda209605a81101d941e2e72dde742c4648dfa71c80bb1c0368" exitCode=0
Jul 28 19:53:13.621517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.621474    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq" event={"ID":"46885940-23b6-4079-8abf-92ce1a09bde1","Type":"ContainerDied","Data":"513ff881fd3d0dda209605a81101d941e2e72dde742c4648dfa71c80bb1c0368"}
Jul 28 19:53:13.814257 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.814231    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:13.964341 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.964225    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgqv8\" (UniqueName: \"kubernetes.io/projected/24f3f95e-89a0-4603-9082-e96fc7f35112-kube-api-access-bgqv8\") pod \"24f3f95e-89a0-4603-9082-e96fc7f35112\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") "
Jul 28 19:53:13.964341 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.964328    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-bundle\") pod \"24f3f95e-89a0-4603-9082-e96fc7f35112\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") "
Jul 28 19:53:13.964570 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.964415    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-util\") pod \"24f3f95e-89a0-4603-9082-e96fc7f35112\" (UID: \"24f3f95e-89a0-4603-9082-e96fc7f35112\") "
Jul 28 19:53:13.964961 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.964929    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-bundle" (OuterVolumeSpecName: "bundle") pod "24f3f95e-89a0-4603-9082-e96fc7f35112" (UID: "24f3f95e-89a0-4603-9082-e96fc7f35112"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:13.966641 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.966618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f3f95e-89a0-4603-9082-e96fc7f35112-kube-api-access-bgqv8" (OuterVolumeSpecName: "kube-api-access-bgqv8") pod "24f3f95e-89a0-4603-9082-e96fc7f35112" (UID: "24f3f95e-89a0-4603-9082-e96fc7f35112"). InnerVolumeSpecName "kube-api-access-bgqv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:53:13.972543 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:13.972501    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-util" (OuterVolumeSpecName: "util") pod "24f3f95e-89a0-4603-9082-e96fc7f35112" (UID: "24f3f95e-89a0-4603-9082-e96fc7f35112"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:14.065236 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.065195    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:14.065236 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.065226    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24f3f95e-89a0-4603-9082-e96fc7f35112-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:14.065236 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.065235    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgqv8\" (UniqueName: \"kubernetes.io/projected/24f3f95e-89a0-4603-9082-e96fc7f35112-kube-api-access-bgqv8\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:14.627818 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.627781    2578 generic.go:356] "Generic (PLEG): container finished" podID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerID="d8c31570446c800619359f9fcd9acd71330850d116a2d6f22dbc6667e10df348" exitCode=0
Jul 28 19:53:14.628442 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.627875    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq" event={"ID":"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91","Type":"ContainerDied","Data":"d8c31570446c800619359f9fcd9acd71330850d116a2d6f22dbc6667e10df348"}
Jul 28 19:53:14.629796 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.629775    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd"
Jul 28 19:53:14.629896 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.629800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd" event={"ID":"24f3f95e-89a0-4603-9082-e96fc7f35112","Type":"ContainerDied","Data":"5bf1742fc1a163e4c255fd04b19a40e9af3eea438c7ceeb90783c68fc12709a2"}
Jul 28 19:53:14.629896 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.629849    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bf1742fc1a163e4c255fd04b19a40e9af3eea438c7ceeb90783c68fc12709a2"
Jul 28 19:53:14.840636 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.840609    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:14.843511 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.843492    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:14.971777 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.971692    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-util\") pod \"46885940-23b6-4079-8abf-92ce1a09bde1\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") "
Jul 28 19:53:14.971777 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.971742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r2ts2\" (UniqueName: \"kubernetes.io/projected/46885940-23b6-4079-8abf-92ce1a09bde1-kube-api-access-r2ts2\") pod \"46885940-23b6-4079-8abf-92ce1a09bde1\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") "
Jul 28 19:53:14.971777 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.971768    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-bundle\") pod \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") "
Jul 28 19:53:14.972069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.971796    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6zjh\" (UniqueName: \"kubernetes.io/projected/5aec8309-bef8-4833-918b-fc5d8f7f14d6-kube-api-access-f6zjh\") pod \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") "
Jul 28 19:53:14.972069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.971821    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-util\") pod \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\" (UID: \"5aec8309-bef8-4833-918b-fc5d8f7f14d6\") "
Jul 28 19:53:14.972069 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.971896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-bundle\") pod \"46885940-23b6-4079-8abf-92ce1a09bde1\" (UID: \"46885940-23b6-4079-8abf-92ce1a09bde1\") "
Jul 28 19:53:14.972461 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.972427    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-bundle" (OuterVolumeSpecName: "bundle") pod "5aec8309-bef8-4833-918b-fc5d8f7f14d6" (UID: "5aec8309-bef8-4833-918b-fc5d8f7f14d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:14.972625 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.972568    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-bundle" (OuterVolumeSpecName: "bundle") pod "46885940-23b6-4079-8abf-92ce1a09bde1" (UID: "46885940-23b6-4079-8abf-92ce1a09bde1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:14.974142 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.974113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46885940-23b6-4079-8abf-92ce1a09bde1-kube-api-access-r2ts2" (OuterVolumeSpecName: "kube-api-access-r2ts2") pod "46885940-23b6-4079-8abf-92ce1a09bde1" (UID: "46885940-23b6-4079-8abf-92ce1a09bde1"). InnerVolumeSpecName "kube-api-access-r2ts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:53:14.974142 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.974129    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aec8309-bef8-4833-918b-fc5d8f7f14d6-kube-api-access-f6zjh" (OuterVolumeSpecName: "kube-api-access-f6zjh") pod "5aec8309-bef8-4833-918b-fc5d8f7f14d6" (UID: "5aec8309-bef8-4833-918b-fc5d8f7f14d6"). InnerVolumeSpecName "kube-api-access-f6zjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:53:14.977724 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.977690    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-util" (OuterVolumeSpecName: "util") pod "46885940-23b6-4079-8abf-92ce1a09bde1" (UID: "46885940-23b6-4079-8abf-92ce1a09bde1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:14.977907 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:14.977886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-util" (OuterVolumeSpecName: "util") pod "5aec8309-bef8-4833-918b-fc5d8f7f14d6" (UID: "5aec8309-bef8-4833-918b-fc5d8f7f14d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:15.073389 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.073357    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:15.073389 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.073383    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6zjh\" (UniqueName: \"kubernetes.io/projected/5aec8309-bef8-4833-918b-fc5d8f7f14d6-kube-api-access-f6zjh\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:15.073389 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.073395    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5aec8309-bef8-4833-918b-fc5d8f7f14d6-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:15.073608 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.073404    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:15.073608 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.073412    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/46885940-23b6-4079-8abf-92ce1a09bde1-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:15.073608 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.073420    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2ts2\" (UniqueName: \"kubernetes.io/projected/46885940-23b6-4079-8abf-92ce1a09bde1-kube-api-access-r2ts2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:15.636517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.636485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm"
Jul 28 19:53:15.636517 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.636503    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm" event={"ID":"5aec8309-bef8-4833-918b-fc5d8f7f14d6","Type":"ContainerDied","Data":"f0da3d08d9afc920e61069ff6a4d29744346511d17d2be157c16cd00f0f767fe"}
Jul 28 19:53:15.636982 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.636576    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0da3d08d9afc920e61069ff6a4d29744346511d17d2be157c16cd00f0f767fe"
Jul 28 19:53:15.638302 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.638282    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq"
Jul 28 19:53:15.638444 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.638281    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq" event={"ID":"46885940-23b6-4079-8abf-92ce1a09bde1","Type":"ContainerDied","Data":"3bb499ea348f1cdb96fa313cc2a0eebd542899c361672468fa2e64be29a1a3bf"}
Jul 28 19:53:15.638444 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.638412    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bb499ea348f1cdb96fa313cc2a0eebd542899c361672468fa2e64be29a1a3bf"
Jul 28 19:53:15.640536 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.640490    2578 generic.go:356] "Generic (PLEG): container finished" podID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerID="6672b1d26c87780ecc46b3e39cd5f992a1400f2d48078b0429f5d089ddbc5ac7" exitCode=0
Jul 28 19:53:15.640623 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:15.640561    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq" event={"ID":"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91","Type":"ContainerDied","Data":"6672b1d26c87780ecc46b3e39cd5f992a1400f2d48078b0429f5d089ddbc5ac7"}
Jul 28 19:53:16.776280 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.776248    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:16.887910 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.887872    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfsmf\" (UniqueName: \"kubernetes.io/projected/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-kube-api-access-zfsmf\") pod \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") "
Jul 28 19:53:16.888100 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.887984    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-bundle\") pod \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") "
Jul 28 19:53:16.888100 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.888037    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-util\") pod \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\" (UID: \"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91\") "
Jul 28 19:53:16.888541 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.888485    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-bundle" (OuterVolumeSpecName: "bundle") pod "d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" (UID: "d04bb88d-00c0-46fd-bc1d-0138d0ea3f91"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:16.890103 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.890082    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-kube-api-access-zfsmf" (OuterVolumeSpecName: "kube-api-access-zfsmf") pod "d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" (UID: "d04bb88d-00c0-46fd-bc1d-0138d0ea3f91"). InnerVolumeSpecName "kube-api-access-zfsmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:53:16.922447 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.922360    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-util" (OuterVolumeSpecName: "util") pod "d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" (UID: "d04bb88d-00c0-46fd-bc1d-0138d0ea3f91"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 19:53:16.988982 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.988949    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfsmf\" (UniqueName: \"kubernetes.io/projected/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-kube-api-access-zfsmf\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:16.988982 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.988978    2578 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:16.989157 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:16.988989    2578 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d04bb88d-00c0-46fd-bc1d-0138d0ea3f91-util\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:53:17.652287 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:17.652229    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq"
Jul 28 19:53:17.652287 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:17.652228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq" event={"ID":"d04bb88d-00c0-46fd-bc1d-0138d0ea3f91","Type":"ContainerDied","Data":"c001b0e60782333229dc1a07a3abb34a1a65de79e6f846c44def7fe99ad5a5c5"}
Jul 28 19:53:17.652554 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:17.652321    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c001b0e60782333229dc1a07a3abb34a1a65de79e6f846c44def7fe99ad5a5c5"
Jul 28 19:53:39.223383 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:53:39.223345    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbc95699-sbbrr"]
Jul 28 19:54:04.245332 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.245216    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fbc95699-sbbrr" podUID="83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" containerName="console" containerID="cri-o://abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f" gracePeriod=15
Jul 28 19:54:04.499734 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.499676    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:54:04.604474 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604437    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-service-ca\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.604474 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604475    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-trusted-ca-bundle\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.604684 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604505    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlmf5\" (UniqueName: \"kubernetes.io/projected/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-kube-api-access-dlmf5\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.604684 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604537    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-config\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.604684 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604564    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-oauth-config\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.604684 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-serving-cert\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.604684 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604620    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-oauth-serving-cert\") pod \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\" (UID: \"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b\") "
Jul 28 19:54:04.605040 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.604903    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-service-ca" (OuterVolumeSpecName: "service-ca") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:54:04.605187 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.605008    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:54:04.605187 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.605113    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:54:04.605187 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.605109    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-config" (OuterVolumeSpecName: "console-config") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 19:54:04.606892 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.606861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:54:04.606985 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.606900    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-kube-api-access-dlmf5" (OuterVolumeSpecName: "kube-api-access-dlmf5") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "kube-api-access-dlmf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 19:54:04.606985 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.606933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" (UID: "83e368a7-cc13-43f0-b3b5-de7f65b2ea7b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 19:54:04.705497 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705456    2578 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-oauth-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.705497 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705489    2578 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.705497 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705499    2578 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-oauth-serving-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.705497 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705508    2578 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.705783 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705517    2578 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-trusted-ca-bundle\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.705783 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705526    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlmf5\" (UniqueName: \"kubernetes.io/projected/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-kube-api-access-dlmf5\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.705783 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.705534    2578 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b-console-config\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 19:54:04.881748 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.881712    2578 generic.go:356] "Generic (PLEG): container finished" podID="83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" containerID="abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f" exitCode=0
Jul 28 19:54:04.881920 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.881761    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbc95699-sbbrr"
Jul 28 19:54:04.881920 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.881801    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbc95699-sbbrr" event={"ID":"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b","Type":"ContainerDied","Data":"abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f"}
Jul 28 19:54:04.881920 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.881853    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbc95699-sbbrr" event={"ID":"83e368a7-cc13-43f0-b3b5-de7f65b2ea7b","Type":"ContainerDied","Data":"21f399ef6dc8591cefab270d00c285830a01b545c63e8da31f0d20a437176b0b"}
Jul 28 19:54:04.881920 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.881873    2578 scope.go:117] "RemoveContainer" containerID="abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f"
Jul 28 19:54:04.890916 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.890898    2578 scope.go:117] "RemoveContainer" containerID="abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f"
Jul 28 19:54:04.891170 ip-10-0-133-160 kubenswrapper[2578]: E0728 19:54:04.891152    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f\": container with ID starting with abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f not found: ID does not exist" containerID="abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f"
Jul 28 19:54:04.891227 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.891177    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f"} err="failed to get container status \"abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f\": rpc error: code = NotFound desc = could not find container \"abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f\": container with ID starting with abf2d0f3d84ce7365fb1e9fb602baa521ec3ad8fc9439daa18ce54d462ff967f not found: ID does not exist"
Jul 28 19:54:04.912865 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.912839    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbc95699-sbbrr"]
Jul 28 19:54:04.917539 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:04.917518    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbc95699-sbbrr"]
Jul 28 19:54:05.651188 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:05.651141    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" path="/var/lib/kubelet/pods/83e368a7-cc13-43f0-b3b5-de7f65b2ea7b/volumes"
Jul 28 19:54:23.788611 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788575    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788857    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788868    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788878    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788883    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788892    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788898    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788908    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788912    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788921    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788926    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788932    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788937    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788942    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788946    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788953    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788958    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788963    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" containerName="console"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788968    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" containerName="console"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788975    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788981    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788986    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788991    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="pull"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.788999    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789003    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="util"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789011    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="extract"
Jul 28 19:54:23.789036 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789016    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="extract"
Jul 28 19:54:23.789817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789061    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="46885940-23b6-4079-8abf-92ce1a09bde1" containerName="extract"
Jul 28 19:54:23.789817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789070    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f3f95e-89a0-4603-9082-e96fc7f35112" containerName="extract"
Jul 28 19:54:23.789817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789077    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aec8309-bef8-4833-918b-fc5d8f7f14d6" containerName="extract"
Jul 28 19:54:23.789817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789082    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e368a7-cc13-43f0-b3b5-de7f65b2ea7b" containerName="console"
Jul 28 19:54:23.789817 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.789088    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d04bb88d-00c0-46fd-bc1d-0138d0ea3f91" containerName="extract"
Jul 28 19:54:23.793452 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.793433    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:23.795883 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.795862    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 19:54:23.795973 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.795930    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mflq8\""
Jul 28 19:54:23.795973 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.795953    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 28 19:54:23.797211 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.797181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 19:54:23.800001 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.799978    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:54:23.832729 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.832699    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:54:23.962710 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.962671    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hmjh\" (UniqueName: \"kubernetes.io/projected/c2cba7f5-e09b-46ec-8370-70fa21301873-kube-api-access-8hmjh\") pod \"limitador-limitador-54d6d988d5-nvnk2\" (UID: \"c2cba7f5-e09b-46ec-8370-70fa21301873\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:23.962880 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:23.962801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2cba7f5-e09b-46ec-8370-70fa21301873-config-file\") pod \"limitador-limitador-54d6d988d5-nvnk2\" (UID: \"c2cba7f5-e09b-46ec-8370-70fa21301873\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:24.064104 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.064007    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2cba7f5-e09b-46ec-8370-70fa21301873-config-file\") pod \"limitador-limitador-54d6d988d5-nvnk2\" (UID: \"c2cba7f5-e09b-46ec-8370-70fa21301873\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:24.064104 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.064100    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hmjh\" (UniqueName: \"kubernetes.io/projected/c2cba7f5-e09b-46ec-8370-70fa21301873-kube-api-access-8hmjh\") pod \"limitador-limitador-54d6d988d5-nvnk2\" (UID: \"c2cba7f5-e09b-46ec-8370-70fa21301873\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:24.064739 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.064719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c2cba7f5-e09b-46ec-8370-70fa21301873-config-file\") pod \"limitador-limitador-54d6d988d5-nvnk2\" (UID: \"c2cba7f5-e09b-46ec-8370-70fa21301873\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:24.079390 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.079361    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hmjh\" (UniqueName: \"kubernetes.io/projected/c2cba7f5-e09b-46ec-8370-70fa21301873-kube-api-access-8hmjh\") pod \"limitador-limitador-54d6d988d5-nvnk2\" (UID: \"c2cba7f5-e09b-46ec-8370-70fa21301873\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:24.104791 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.104751    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:24.243901 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.243873    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:54:24.246073 ip-10-0-133-160 kubenswrapper[2578]: W0728 19:54:24.246045    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2cba7f5_e09b_46ec_8370_70fa21301873.slice/crio-2745b78533a9962625618061cc42690712bfbd849bdcdef778e9acdc1e281110 WatchSource:0}: Error finding container 2745b78533a9962625618061cc42690712bfbd849bdcdef778e9acdc1e281110: Status 404 returned error can't find the container with id 2745b78533a9962625618061cc42690712bfbd849bdcdef778e9acdc1e281110
Jul 28 19:54:24.982987 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:24.982949    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2" event={"ID":"c2cba7f5-e09b-46ec-8370-70fa21301873","Type":"ContainerStarted","Data":"2745b78533a9962625618061cc42690712bfbd849bdcdef778e9acdc1e281110"}
Jul 28 19:54:27.999225 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:27.999185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2" event={"ID":"c2cba7f5-e09b-46ec-8370-70fa21301873","Type":"ContainerStarted","Data":"bef95dbd344b78defc346b24e3a1f084dca4def1c92dc6d9d7bf350187ac916b"}
Jul 28 19:54:27.999638 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:27.999244    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:54:28.020661 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:28.020604    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2" podStartSLOduration=2.3210482470000002 podStartE2EDuration="5.020588496s" totalImagesPullingTime="2.699540249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 19:54:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 19:54:28.018070326 +0000 UTC m=+470.918287169" watchObservedRunningTime="2026-07-28 19:54:28.020588496 +0000 UTC m=+470.920805363"
Jul 28 19:54:39.005526 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:54:39.005494    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"
Jul 28 19:55:39.234831 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:55:39.234751    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:56:10.297716 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:56:10.297674    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:56:23.581644 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:56:23.581610    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:56:37.519633 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:56:37.519603    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:56:37.521133 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:56:37.521110    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 19:56:39.371946 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:56:39.371908    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:56:45.465744 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:56:45.465711    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:57:14.662628 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:57:14.662534    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:57:23.865611 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:57:23.865572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:58:46.364286 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:58:46.364185    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:58:55.359066 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:58:55.359031    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 19:59:06.072572 ip-10-0-133-160 kubenswrapper[2578]: I0728 19:59:06.072530    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:00:00.152026 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.151944    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"]
Jul 28 20:00:00.155062 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.155044    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"
Jul 28 20:00:00.157751 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.157726    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zkwl\""
Jul 28 20:00:00.163011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.162983    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"]
Jul 28 20:00:00.165716 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.165693    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"
Jul 28 20:00:00.294341 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.294318    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"]
Jul 28 20:00:00.296205 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:00:00.296175    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc6896dc_5b67_47d0_b1d4_a73e104712d2.slice/crio-55cff40bdf2723c458ff14562ae0179805d6e74991d339b1c7b50dcc3da23fbd WatchSource:0}: Error finding container 55cff40bdf2723c458ff14562ae0179805d6e74991d339b1c7b50dcc3da23fbd: Status 404 returned error can't find the container with id 55cff40bdf2723c458ff14562ae0179805d6e74991d339b1c7b50dcc3da23fbd
Jul 28 20:00:00.298034 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.298020    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:00:00.594069 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:00.594033    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6" event={"ID":"fc6896dc-5b67-47d0-b1d4-a73e104712d2","Type":"ContainerStarted","Data":"55cff40bdf2723c458ff14562ae0179805d6e74991d339b1c7b50dcc3da23fbd"}
Jul 28 20:00:09.639347 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:09.639307    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6" event={"ID":"fc6896dc-5b67-47d0-b1d4-a73e104712d2","Type":"ContainerStarted","Data":"e8746cc0b015187426cb0bee38abe5950d0c3eb7b8e2f9a469622dc92364e914"}
Jul 28 20:00:09.656112 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:09.656049    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6" podStartSLOduration=1.051084029 podStartE2EDuration="9.65603299s" totalImagesPullingTime="8.604948961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:00:09.655345654 +0000 UTC m=+812.555562498" watchObservedRunningTime="2026-07-28 20:00:09.65603299 +0000 UTC m=+812.556249832"
Jul 28 20:00:10.646283 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:10.646238    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc6896dc-5b67-47d0-b1d4-a73e104712d2" containerID="e8746cc0b015187426cb0bee38abe5950d0c3eb7b8e2f9a469622dc92364e914" exitCode=0
Jul 28 20:00:10.646751 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:10.646324    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6" event={"ID":"fc6896dc-5b67-47d0-b1d4-a73e104712d2","Type":"ContainerDied","Data":"e8746cc0b015187426cb0bee38abe5950d0c3eb7b8e2f9a469622dc92364e914"}
Jul 28 20:00:11.778959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:11.778934    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"
Jul 28 20:00:12.657871 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:12.657840    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6"
Jul 28 20:00:12.658073 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:12.657884    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754480-q46d6" event={"ID":"fc6896dc-5b67-47d0-b1d4-a73e104712d2","Type":"ContainerDied","Data":"55cff40bdf2723c458ff14562ae0179805d6e74991d339b1c7b50dcc3da23fbd"}
Jul 28 20:00:12.658073 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:12.657928    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55cff40bdf2723c458ff14562ae0179805d6e74991d339b1c7b50dcc3da23fbd"
Jul 28 20:00:47.661970 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:47.661931    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:00:54.969988 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:00:54.969951    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:01:31.266955 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:01:31.266871    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:01:37.550334 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:01:37.550306    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:01:37.553330 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:01:37.553309    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:01:41.279770 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:01:41.279733    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:01:50.478251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:01:50.478212    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:02:01.589157 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:02:01.589119    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:02:11.883932 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:02:11.883897    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:02:23.269056 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:02:23.269020    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:02:34.068242 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:02:34.068206    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:02:43.867922 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:02:43.867878    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:03:09.678970 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:03:09.678874    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:03:19.172281 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:03:19.172225    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:03:42.968232 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:03:42.968199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:03:53.079095 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:03:53.079058    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:04:02.867823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:04:02.867782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:04:18.672629 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:04:18.672586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:04:28.173722 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:04:28.173680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:04:37.388191 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:04:37.388099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:04:46.679562 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:04:46.679521    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:04:55.878784 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:04:55.878750    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:05:07.876286 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:05:07.876230    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:05:15.876559 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:05:15.876524    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:06:37.583614 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:06:37.583579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:06:37.587225 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:06:37.587193    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:06:54.967705 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:06:54.967664    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:07:05.669922 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:07:05.669885    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:07:18.881278 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:07:18.881224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:07:28.768744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:07:28.768705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:07:39.173887 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:07:39.173843    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:07:49.879320 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:07:49.879260    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:07:59.685180 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:07:59.685133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:08:12.696144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:08:12.696099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:08:21.275542 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:08:21.275493    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:08:31.273976 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:08:31.273932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:08:40.969733 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:08:40.969690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:09:03.980073 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:09:03.979983    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:09:13.875480 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:09:13.875437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:09:24.972695 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:09:24.972659    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:09:38.169351 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:09:38.169314    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:09:55.266929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:09:55.266886    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:10:08.869283 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:10:08.869238    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:10:17.870257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:10:17.870221    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:10:29.305991 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:10:29.305894    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:10:38.474048 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:10:38.474012    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:10:47.375649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:10:47.375611    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:10:55.881768 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:10:55.881737    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:11:06.872332 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:06.872295    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:11:15.873297 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:15.873247    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:11:29.073213 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:29.073176    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:11:37.621598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:37.621570    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:11:37.627410 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:37.627384    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:11:37.871566 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:37.871534    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:11:47.175762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:47.175724    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:11:59.773727 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:11:59.773638    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:12:08.912090 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:12:08.912055    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:12:16.273285 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:12:16.273224    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:12:30.482845 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:12:30.482807    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:12:42.372860 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:12:42.372822    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:12:52.576175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:12:52.576141    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:13:02.075897 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:02.075858    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:13:10.777332 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:10.777288    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:13:21.388160 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:21.388120    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:13:33.026299 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:33.026189    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:13:39.257510 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.257471    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"]
Jul 28 20:13:39.257896 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.257782    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc6896dc-5b67-47d0-b1d4-a73e104712d2" containerName="cleanup"
Jul 28 20:13:39.257896 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.257793    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc6896dc-5b67-47d0-b1d4-a73e104712d2" containerName="cleanup"
Jul 28 20:13:39.257896 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.257860    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc6896dc-5b67-47d0-b1d4-a73e104712d2" containerName="cleanup"
Jul 28 20:13:39.260925 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.260906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.264899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.264872    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0603e8fc-openshift-default-dockercfg-x7plb\""
Jul 28 20:13:39.278488 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278450    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-envoy\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278488 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278493    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-data\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278520    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-socket\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-istio-token\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278631    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94e35c5-4410-4953-8133-bc687a23ce97-istio-podinfo\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278659    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-certs\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhnpq\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-kube-api-access-dhnpq\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278852 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278752    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-credential-socket\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.278852 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.278776    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94e35c5-4410-4953-8133-bc687a23ce97-istiod-ca-cert\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.311241 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.311199    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"]
Jul 28 20:13:39.379467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94e35c5-4410-4953-8133-bc687a23ce97-istiod-ca-cert\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379475    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-envoy\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379491    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-data\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379525    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-socket\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379559    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-istio-token\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379595    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94e35c5-4410-4953-8133-bc687a23ce97-istio-podinfo\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379622    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-certs\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379643    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhnpq\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-kube-api-access-dhnpq\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.379737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.379703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-credential-socket\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.380109 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.380084    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-data\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.380180 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.380113    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-credential-socket\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.380242 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.380202    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-certs\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.380318 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.380259    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-socket\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.380492 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.380476    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94e35c5-4410-4953-8133-bc687a23ce97-istiod-ca-cert\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.382342 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.382316    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-envoy\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.382463 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.382446    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94e35c5-4410-4953-8133-bc687a23ce97-istio-podinfo\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.391319 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.391254    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-istio-token\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.392761 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.392731    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhnpq\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-kube-api-access-dhnpq\") pod \"e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") " pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.571397 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.571293    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:39.725345 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.725317    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"]
Jul 28 20:13:39.727460 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:13:39.727421    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd94e35c5_4410_4953_8133_bc687a23ce97.slice/crio-241ca2e73c6462a2581a860225558a53a72b702cc0fd702a78972f721f392f19 WatchSource:0}: Error finding container 241ca2e73c6462a2581a860225558a53a72b702cc0fd702a78972f721f392f19: Status 404 returned error can't find the container with id 241ca2e73c6462a2581a860225558a53a72b702cc0fd702a78972f721f392f19
Jul 28 20:13:39.729449 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:39.729433    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:13:40.569147 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:40.569106    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" event={"ID":"d94e35c5-4410-4953-8133-bc687a23ce97","Type":"ContainerStarted","Data":"241ca2e73c6462a2581a860225558a53a72b702cc0fd702a78972f721f392f19"}
Jul 28 20:13:44.159993 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:44.159951    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:13:44.160284 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:44.160026    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:13:44.160284 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:44.160055    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:13:44.592093 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:44.592048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" event={"ID":"d94e35c5-4410-4953-8133-bc687a23ce97","Type":"ContainerStarted","Data":"c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"}
Jul 28 20:13:44.614598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:44.614526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" podStartSLOduration=1.184417163 podStartE2EDuration="5.614509676s" totalImagesPullingTime="4.430092513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:44.612902195 +0000 UTC m=+1627.513119040" watchObservedRunningTime="2026-07-28 20:13:44.614509676 +0000 UTC m=+1627.514726517"
Jul 28 20:13:45.571752 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:45.571715    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:45.576759 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:45.576730    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:45.596744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:45.596698    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:45.597690 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:45.597660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:13:47.012792 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.012751    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"]
Jul 28 20:13:47.016334 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.016314    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.019102 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.019073    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0603e8fc\""
Jul 28 20:13:47.019218 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.019073    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0603e8fc-dockercfg-2w8r9\""
Jul 28 20:13:47.027007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.026980    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"]
Jul 28 20:13:47.045137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.045100    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b74cc4b-1a78-4f45-934d-c43ea47d7225-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.045347 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.045154    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl94t\" (UniqueName: \"kubernetes.io/projected/3b74cc4b-1a78-4f45-934d-c43ea47d7225-kube-api-access-kl94t\") pod \"payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.146386 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.146349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b74cc4b-1a78-4f45-934d-c43ea47d7225-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.146557 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.146396    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl94t\" (UniqueName: \"kubernetes.io/projected/3b74cc4b-1a78-4f45-934d-c43ea47d7225-kube-api-access-kl94t\") pod \"payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.147154 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.147135    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b74cc4b-1a78-4f45-934d-c43ea47d7225-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.157260 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.157239    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl94t\" (UniqueName: \"kubernetes.io/projected/3b74cc4b-1a78-4f45-934d-c43ea47d7225-kube-api-access-kl94t\") pod \"payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.327393 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.327258    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:47.466624 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.466588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"]
Jul 28 20:13:47.468500 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:13:47.468471    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b74cc4b_1a78_4f45_934d_c43ea47d7225.slice/crio-0545c494b83730aec51fd213efecb07308a736229707c2b12d61df8dd753c8df WatchSource:0}: Error finding container 0545c494b83730aec51fd213efecb07308a736229707c2b12d61df8dd753c8df: Status 404 returned error can't find the container with id 0545c494b83730aec51fd213efecb07308a736229707c2b12d61df8dd753c8df
Jul 28 20:13:47.612198 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:47.612099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt" event={"ID":"3b74cc4b-1a78-4f45-934d-c43ea47d7225","Type":"ContainerStarted","Data":"0545c494b83730aec51fd213efecb07308a736229707c2b12d61df8dd753c8df"}
Jul 28 20:13:50.630248 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:50.630206    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt" event={"ID":"3b74cc4b-1a78-4f45-934d-c43ea47d7225","Type":"ContainerStarted","Data":"0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec"}
Jul 28 20:13:50.630679 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:50.630296    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:13:50.650051 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:50.649993    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt" podStartSLOduration=2.54021603 podStartE2EDuration="4.64997592s" totalImagesPullingTime="2.10975989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:13:50.647919197 +0000 UTC m=+1633.548136049" watchObservedRunningTime="2026-07-28 20:13:50.64997592 +0000 UTC m=+1633.550192740"
Jul 28 20:13:50.770078 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:50.770045    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:13:58.677371 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:58.677283    2578 generic.go:356] "Generic (PLEG): container finished" podID="d94e35c5-4410-4953-8133-bc687a23ce97" containerID="c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112" exitCode=0
Jul 28 20:13:58.677371 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:58.677343    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" event={"ID":"d94e35c5-4410-4953-8133-bc687a23ce97","Type":"ContainerDied","Data":"c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"}
Jul 28 20:13:58.677809 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:58.677750    2578 scope.go:117] "RemoveContainer" containerID="c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"
Jul 28 20:13:58.679729 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:58.679696    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:13:58.679804 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:58.679788    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:13:58.679850 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:58.679838    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:13:59.684852 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:13:59.684812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" event={"ID":"d94e35c5-4410-4953-8133-bc687a23ce97","Type":"ContainerStarted","Data":"461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9"}
Jul 28 20:14:00.572133 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:00.572091    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:14:00.573385 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:00.573357    2578 patch_prober.go:28] interesting pod/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 28 20:14:00.573507 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:00.573419    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 28 20:14:00.596971 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:00.596930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:14:01.572631 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:01.572591    2578 patch_prober.go:28] interesting pod/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 28 20:14:01.573051 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:01.572652    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 28 20:14:01.641208 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:01.641177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:14:02.572119 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:02.572082    2578 patch_prober.go:28] interesting pod/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Jul 28 20:14:02.572349 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:02.572150    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Jul 28 20:14:03.575598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:03.575554    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:14:03.576487 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:03.576469    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:14:11.593376 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.593335    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"]
Jul 28 20:14:11.596650 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.596630    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:11.599286 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.599252    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:14:11.599404 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.599250    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zkwl\""
Jul 28 20:14:11.610173 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.610103    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"]
Jul 28 20:14:11.674620 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.674573    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89680044-03a2-4eb6-9353-07faac56e957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h\" (UID: \"89680044-03a2-4eb6-9353-07faac56e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:11.775390 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.775341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89680044-03a2-4eb6-9353-07faac56e957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h\" (UID: \"89680044-03a2-4eb6-9353-07faac56e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:11.775990 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.775966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89680044-03a2-4eb6-9353-07faac56e957-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h\" (UID: \"89680044-03a2-4eb6-9353-07faac56e957\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:11.909602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:11.909488    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:12.048775 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:12.048747    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"]
Jul 28 20:14:12.051187 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:12.051156    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89680044_03a2_4eb6_9353_07faac56e957.slice/crio-41e6568a05181533ffc28c555a6dc0d9638bd48e7de05b2b1310fb95368bc472 WatchSource:0}: Error finding container 41e6568a05181533ffc28c555a6dc0d9638bd48e7de05b2b1310fb95368bc472: Status 404 returned error can't find the container with id 41e6568a05181533ffc28c555a6dc0d9638bd48e7de05b2b1310fb95368bc472
Jul 28 20:14:12.754171 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:12.754138    2578 generic.go:356] "Generic (PLEG): container finished" podID="89680044-03a2-4eb6-9353-07faac56e957" containerID="1c98abcabf702a04d9f14d7f2aa4921b7703c4a2e064bb23e06495a60751a631" exitCode=0
Jul 28 20:14:12.754616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:12.754228    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h" event={"ID":"89680044-03a2-4eb6-9353-07faac56e957","Type":"ContainerDied","Data":"1c98abcabf702a04d9f14d7f2aa4921b7703c4a2e064bb23e06495a60751a631"}
Jul 28 20:14:12.754616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:12.754302    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h" event={"ID":"89680044-03a2-4eb6-9353-07faac56e957","Type":"ContainerStarted","Data":"41e6568a05181533ffc28c555a6dc0d9638bd48e7de05b2b1310fb95368bc472"}
Jul 28 20:14:13.891525 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:13.891493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:13.992054 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:13.992011    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89680044-03a2-4eb6-9353-07faac56e957-maas-api-service-ca\") pod \"89680044-03a2-4eb6-9353-07faac56e957\" (UID: \"89680044-03a2-4eb6-9353-07faac56e957\") "
Jul 28 20:14:13.992428 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:13.992405    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89680044-03a2-4eb6-9353-07faac56e957-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89680044-03a2-4eb6-9353-07faac56e957" (UID: "89680044-03a2-4eb6-9353-07faac56e957"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:14.093682 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:14.093582    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89680044-03a2-4eb6-9353-07faac56e957-maas-api-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:14.765385 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:14.765356    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"
Jul 28 20:14:14.765564 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:14.765379    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h" event={"ID":"89680044-03a2-4eb6-9353-07faac56e957","Type":"ContainerDied","Data":"41e6568a05181533ffc28c555a6dc0d9638bd48e7de05b2b1310fb95368bc472"}
Jul 28 20:14:14.765564 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:14.765427    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41e6568a05181533ffc28c555a6dc0d9638bd48e7de05b2b1310fb95368bc472"
Jul 28 20:14:16.910021 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:16.909976    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"]
Jul 28 20:14:16.910517 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:16.910290    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt" podUID="3b74cc4b-1a78-4f45-934d-c43ea47d7225" containerName="payload-pre-processing" containerID="cri-o://0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec" gracePeriod=30
Jul 28 20:14:17.047571 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.047527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"]
Jul 28 20:14:17.052734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.052698    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0603e8fc-kp97h"]
Jul 28 20:14:17.179844 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.179662    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:14:17.182135 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.182104    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"]
Jul 28 20:14:17.182364 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.182340    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy" containerID="cri-o://461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9" gracePeriod=30
Jul 28 20:14:17.323231 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.323190    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kl94t\" (UniqueName: \"kubernetes.io/projected/3b74cc4b-1a78-4f45-934d-c43ea47d7225-kube-api-access-kl94t\") pod \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") "
Jul 28 20:14:17.323412 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.323343    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b74cc4b-1a78-4f45-934d-c43ea47d7225-plugins-config-volume\") pod \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\" (UID: \"3b74cc4b-1a78-4f45-934d-c43ea47d7225\") "
Jul 28 20:14:17.323722 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.323700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b74cc4b-1a78-4f45-934d-c43ea47d7225-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b74cc4b-1a78-4f45-934d-c43ea47d7225" (UID: "3b74cc4b-1a78-4f45-934d-c43ea47d7225"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:17.325945 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.325912    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b74cc4b-1a78-4f45-934d-c43ea47d7225-kube-api-access-kl94t" (OuterVolumeSpecName: "kube-api-access-kl94t") pod "3b74cc4b-1a78-4f45-934d-c43ea47d7225" (UID: "3b74cc4b-1a78-4f45-934d-c43ea47d7225"). InnerVolumeSpecName "kube-api-access-kl94t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:17.424965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.424863    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b74cc4b-1a78-4f45-934d-c43ea47d7225-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:17.424965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.424899    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl94t\" (UniqueName: \"kubernetes.io/projected/3b74cc4b-1a78-4f45-934d-c43ea47d7225-kube-api-access-kl94t\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:17.652941 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.652688    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89680044-03a2-4eb6-9353-07faac56e957" path="/var/lib/kubelet/pods/89680044-03a2-4eb6-9353-07faac56e957/volumes"
Jul 28 20:14:17.782702 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.782664    2578 generic.go:356] "Generic (PLEG): container finished" podID="3b74cc4b-1a78-4f45-934d-c43ea47d7225" containerID="0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec" exitCode=0
Jul 28 20:14:17.782856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.782709    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"
Jul 28 20:14:17.782856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.782751    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt" event={"ID":"3b74cc4b-1a78-4f45-934d-c43ea47d7225","Type":"ContainerDied","Data":"0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec"}
Jul 28 20:14:17.782856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.782812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt" event={"ID":"3b74cc4b-1a78-4f45-934d-c43ea47d7225","Type":"ContainerDied","Data":"0545c494b83730aec51fd213efecb07308a736229707c2b12d61df8dd753c8df"}
Jul 28 20:14:17.782856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.782842    2578 scope.go:117] "RemoveContainer" containerID="0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec"
Jul 28 20:14:17.791895 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.791875    2578 scope.go:117] "RemoveContainer" containerID="0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec"
Jul 28 20:14:17.792169 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:17.792148    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec\": container with ID starting with 0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec not found: ID does not exist" containerID="0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec"
Jul 28 20:14:17.792223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.792180    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec"} err="failed to get container status \"0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec\": rpc error: code = NotFound desc = could not find container \"0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec\": container with ID starting with 0a6ec8337d97a6feceafd3e60554fdbcba9cddbc02f75d72700de615be6ce4ec not found: ID does not exist"
Jul 28 20:14:17.801077 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.801045    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"]
Jul 28 20:14:17.809216 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:17.809191    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0603e8fc-859467d7d4-gpsmt"]
Jul 28 20:14:18.578929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:18.578883    2578 patch_prober.go:28] interesting pod/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 28 20:14:18.579341 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:18.578949    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:14:19.651234 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:19.651197    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b74cc4b-1a78-4f45-934d-c43ea47d7225" path="/var/lib/kubelet/pods/3b74cc4b-1a78-4f45-934d-c43ea47d7225/volumes"
Jul 28 20:14:20.668633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.668583    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:14:20.899373 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899334    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"]
Jul 28 20:14:20.899747 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899729    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b74cc4b-1a78-4f45-934d-c43ea47d7225" containerName="payload-pre-processing"
Jul 28 20:14:20.899835 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899750    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b74cc4b-1a78-4f45-934d-c43ea47d7225" containerName="payload-pre-processing"
Jul 28 20:14:20.899835 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899764    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89680044-03a2-4eb6-9353-07faac56e957" containerName="revoke-keys"
Jul 28 20:14:20.899835 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899771    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="89680044-03a2-4eb6-9353-07faac56e957" containerName="revoke-keys"
Jul 28 20:14:20.899992 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899846    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b74cc4b-1a78-4f45-934d-c43ea47d7225" containerName="payload-pre-processing"
Jul 28 20:14:20.899992 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.899863    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="89680044-03a2-4eb6-9353-07faac56e957" containerName="revoke-keys"
Jul 28 20:14:20.904205 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.904181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:20.907203 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.907182    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:14:20.907388 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.907320    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:14:20.907468 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.907423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-916f7ae5-serving-cert\""
Jul 28 20:14:20.908527 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.908512    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:14:20.916840 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:20.916817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"]
Jul 28 20:14:21.053729 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.053693    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4adbe-1170-49ce-a323-cc88dd62f49c-openshift-service-ca\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.053729 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.053733    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4adbe-1170-49ce-a323-cc88dd62f49c-maas-api-tls\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.053977 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.053766    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25gjp\" (UniqueName: \"kubernetes.io/projected/63c4adbe-1170-49ce-a323-cc88dd62f49c-kube-api-access-25gjp\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.155116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.155072    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4adbe-1170-49ce-a323-cc88dd62f49c-openshift-service-ca\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.155116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.155117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4adbe-1170-49ce-a323-cc88dd62f49c-maas-api-tls\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.155403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.155148    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25gjp\" (UniqueName: \"kubernetes.io/projected/63c4adbe-1170-49ce-a323-cc88dd62f49c-kube-api-access-25gjp\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.155933 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.155910    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4adbe-1170-49ce-a323-cc88dd62f49c-openshift-service-ca\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.157869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.157840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4adbe-1170-49ce-a323-cc88dd62f49c-maas-api-tls\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.166425 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.166397    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25gjp\" (UniqueName: \"kubernetes.io/projected/63c4adbe-1170-49ce-a323-cc88dd62f49c-kube-api-access-25gjp\") pod \"maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.215428 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.215395    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:21.364421 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.364395    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"]
Jul 28 20:14:21.366783 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:21.366745    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c4adbe_1170_49ce_a323_cc88dd62f49c.slice/crio-5ecfe5ef144d8dfa45ed4352c11daf868a174ab0635e023955f27bba413bf034 WatchSource:0}: Error finding container 5ecfe5ef144d8dfa45ed4352c11daf868a174ab0635e023955f27bba413bf034: Status 404 returned error can't find the container with id 5ecfe5ef144d8dfa45ed4352c11daf868a174ab0635e023955f27bba413bf034
Jul 28 20:14:21.546965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.546924    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"]
Jul 28 20:14:21.551366 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.551346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.555005 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.554977    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-916f7ae5\""
Jul 28 20:14:21.555132 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.554977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-916f7ae5-dockercfg-n7l8m\""
Jul 28 20:14:21.558137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.558075    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"]
Jul 28 20:14:21.561893 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.561869    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.571095 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.571068    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"]
Jul 28 20:14:21.573327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.573243    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"]
Jul 28 20:14:21.659349 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.659312    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f2d7a3-57f4-4fa5-8229-0dada07ec963-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.659542 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.659358    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p72zf\" (UniqueName: \"kubernetes.io/projected/9bc2d155-b57f-4279-8c8d-7186c94147d4-kube-api-access-p72zf\") pod \"payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.659542 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.659388    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67gwl\" (UniqueName: \"kubernetes.io/projected/94f2d7a3-57f4-4fa5-8229-0dada07ec963-kube-api-access-67gwl\") pod \"payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.659542 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.659507    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc2d155-b57f-4279-8c8d-7186c94147d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.760730 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.760685    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f2d7a3-57f4-4fa5-8229-0dada07ec963-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.760730 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.760731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p72zf\" (UniqueName: \"kubernetes.io/projected/9bc2d155-b57f-4279-8c8d-7186c94147d4-kube-api-access-p72zf\") pod \"payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.761253 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.760753    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67gwl\" (UniqueName: \"kubernetes.io/projected/94f2d7a3-57f4-4fa5-8229-0dada07ec963-kube-api-access-67gwl\") pod \"payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.761253 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.760800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc2d155-b57f-4279-8c8d-7186c94147d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.761524 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.761503    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc2d155-b57f-4279-8c8d-7186c94147d4-plugins-config-volume\") pod \"payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.761565 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.761529    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f2d7a3-57f4-4fa5-8229-0dada07ec963-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.770646 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.770618    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67gwl\" (UniqueName: \"kubernetes.io/projected/94f2d7a3-57f4-4fa5-8229-0dada07ec963-kube-api-access-67gwl\") pod \"payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:21.770784 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.770685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p72zf\" (UniqueName: \"kubernetes.io/projected/9bc2d155-b57f-4279-8c8d-7186c94147d4-kube-api-access-p72zf\") pod \"payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") " pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.806235 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.806198    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5" event={"ID":"63c4adbe-1170-49ce-a323-cc88dd62f49c","Type":"ContainerStarted","Data":"5ecfe5ef144d8dfa45ed4352c11daf868a174ab0635e023955f27bba413bf034"}
Jul 28 20:14:21.862624 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.862531    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:21.873620 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:21.873589    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:22.033768 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.033705    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"]
Jul 28 20:14:22.039040 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:22.039003    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc2d155_b57f_4279_8c8d_7186c94147d4.slice/crio-e9c055a38349ea355add5e2d202c69bc749ac6b450775e3eea8f21b992f57253 WatchSource:0}: Error finding container e9c055a38349ea355add5e2d202c69bc749ac6b450775e3eea8f21b992f57253: Status 404 returned error can't find the container with id e9c055a38349ea355add5e2d202c69bc749ac6b450775e3eea8f21b992f57253
Jul 28 20:14:22.062021 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.061991    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"]
Jul 28 20:14:22.064723 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:22.064669    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f2d7a3_57f4_4fa5_8229_0dada07ec963.slice/crio-f839a76bd852c0f1d18f57f030a6837e8cf745994d1fefa1d6b7e0d66126ca50 WatchSource:0}: Error finding container f839a76bd852c0f1d18f57f030a6837e8cf745994d1fefa1d6b7e0d66126ca50: Status 404 returned error can't find the container with id f839a76bd852c0f1d18f57f030a6837e8cf745994d1fefa1d6b7e0d66126ca50
Jul 28 20:14:22.466463 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.466424    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:14:22.569229 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569187    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-data\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569433 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94e35c5-4410-4953-8133-bc687a23ce97-istio-podinfo\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569433 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569288    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhnpq\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-kube-api-access-dhnpq\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569433 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569354    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-socket\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569433 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569379    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-istio-token\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569673 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569482    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-credential-socket\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569673 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569516    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-envoy\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569673 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569540    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-certs\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569673 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569588    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94e35c5-4410-4953-8133-bc687a23ce97-istiod-ca-cert\") pod \"d94e35c5-4410-4953-8133-bc687a23ce97\" (UID: \"d94e35c5-4410-4953-8133-bc687a23ce97\") "
Jul 28 20:14:22.569877 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569801    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:14:22.569932 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.569901    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:14:22.570144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.570118    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d94e35c5-4410-4953-8133-bc687a23ce97-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:22.570226 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.570124    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:14:22.570226 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.570179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-data" (OuterVolumeSpecName: "istio-data") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:14:22.572371 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.572313    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-istio-token" (OuterVolumeSpecName: "istio-token") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:22.572371 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.572312    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d94e35c5-4410-4953-8133-bc687a23ce97-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:14:22.572849 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.572819    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-kube-api-access-dhnpq" (OuterVolumeSpecName: "kube-api-access-dhnpq") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "kube-api-access-dhnpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:22.573064 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.573039    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d94e35c5-4410-4953-8133-bc687a23ce97" (UID: "d94e35c5-4410-4953-8133-bc687a23ce97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:14:22.670587 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670540    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670587 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670584    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d94e35c5-4410-4953-8133-bc687a23ce97-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670599    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhnpq\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-kube-api-access-dhnpq\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670613    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670625    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d94e35c5-4410-4953-8133-bc687a23ce97-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670637    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670651    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670664    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d94e35c5-4410-4953-8133-bc687a23ce97-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.670830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.670678    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d94e35c5-4410-4953-8133-bc687a23ce97-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:22.814998 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.814957    2578 generic.go:356] "Generic (PLEG): container finished" podID="d94e35c5-4410-4953-8133-bc687a23ce97" containerID="461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9" exitCode=0
Jul 28 20:14:22.814998 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.815009    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"
Jul 28 20:14:22.815557 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.815064    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" event={"ID":"d94e35c5-4410-4953-8133-bc687a23ce97","Type":"ContainerDied","Data":"461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9"}
Jul 28 20:14:22.815557 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.815131    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt" event={"ID":"d94e35c5-4410-4953-8133-bc687a23ce97","Type":"ContainerDied","Data":"241ca2e73c6462a2581a860225558a53a72b702cc0fd702a78972f721f392f19"}
Jul 28 20:14:22.815557 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.815174    2578 scope.go:117] "RemoveContainer" containerID="461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9"
Jul 28 20:14:22.818200 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.818077    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b" event={"ID":"9bc2d155-b57f-4279-8c8d-7186c94147d4","Type":"ContainerStarted","Data":"aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271"}
Jul 28 20:14:22.818200 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.818133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b" event={"ID":"9bc2d155-b57f-4279-8c8d-7186c94147d4","Type":"ContainerStarted","Data":"e9c055a38349ea355add5e2d202c69bc749ac6b450775e3eea8f21b992f57253"}
Jul 28 20:14:22.818200 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.818163    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:22.821164 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.821139    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7" event={"ID":"94f2d7a3-57f4-4fa5-8229-0dada07ec963","Type":"ContainerStarted","Data":"d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e"}
Jul 28 20:14:22.821417 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.821389    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7" event={"ID":"94f2d7a3-57f4-4fa5-8229-0dada07ec963","Type":"ContainerStarted","Data":"f839a76bd852c0f1d18f57f030a6837e8cf745994d1fefa1d6b7e0d66126ca50"}
Jul 28 20:14:22.821537 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.821433    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:22.837656 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.837605    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b" podStartSLOduration=1.8375878110000001 podStartE2EDuration="1.837587811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:22.834864221 +0000 UTC m=+1665.735081064" watchObservedRunningTime="2026-07-28 20:14:22.837587811 +0000 UTC m=+1665.737804670"
Jul 28 20:14:22.849983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.849945    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"]
Jul 28 20:14:22.853764 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.853734    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0603e8fc-openshift-default-fff856cc4-5p7dt"]
Jul 28 20:14:22.869833 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:22.869769    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7" podStartSLOduration=1.869746262 podStartE2EDuration="1.869746262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:22.869042852 +0000 UTC m=+1665.769259698" watchObservedRunningTime="2026-07-28 20:14:22.869746262 +0000 UTC m=+1665.769963096"
Jul 28 20:14:23.471101 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:23.470906    2578 scope.go:117] "RemoveContainer" containerID="c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"
Jul 28 20:14:23.480206 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:23.480184    2578 scope.go:117] "RemoveContainer" containerID="461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9"
Jul 28 20:14:23.480577 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:23.480558    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9\": container with ID starting with 461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9 not found: ID does not exist" containerID="461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9"
Jul 28 20:14:23.480631 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:23.480588    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9"} err="failed to get container status \"461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9\": rpc error: code = NotFound desc = could not find container \"461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9\": container with ID starting with 461ba65d95af220c0af42f54acc263b4b12d7d567550bea5f7384a9a38dc64c9 not found: ID does not exist"
Jul 28 20:14:23.480631 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:23.480609    2578 scope.go:117] "RemoveContainer" containerID="c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"
Jul 28 20:14:23.480883 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:23.480862    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112\": container with ID starting with c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112 not found: ID does not exist" containerID="c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"
Jul 28 20:14:23.480943 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:23.480892    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112"} err="failed to get container status \"c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112\": rpc error: code = NotFound desc = could not find container \"c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112\": container with ID starting with c5916842f962cf51cf1927fb946ac532a730c60972b3d81c39cbdc781decb112 not found: ID does not exist"
Jul 28 20:14:23.651864 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:23.651830    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" path="/var/lib/kubelet/pods/d94e35c5-4410-4953-8133-bc687a23ce97/volumes"
Jul 28 20:14:24.835771 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:24.835730    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5" event={"ID":"63c4adbe-1170-49ce-a323-cc88dd62f49c","Type":"ContainerStarted","Data":"97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b"}
Jul 28 20:14:24.836194 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:24.835794    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:24.853342 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:24.853249    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5" podStartSLOduration=2.043152122 podStartE2EDuration="4.85322785s" totalImagesPullingTime="2.810075728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:24.85254229 +0000 UTC m=+1667.752759156" watchObservedRunningTime="2026-07-28 20:14:24.85322785 +0000 UTC m=+1667.753444693"
Jul 28 20:14:30.846146 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:30.846113    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:31.272381 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:31.272328    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:14:33.830490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:33.830452    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:33.830981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:33.830751    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:45.786574 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.786526    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"]
Jul 28 20:14:45.787184 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.787055    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy"
Jul 28 20:14:45.787184 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.787075    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy"
Jul 28 20:14:45.787184 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.787087    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy"
Jul 28 20:14:45.787184 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.787096    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy"
Jul 28 20:14:45.787184 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.787184    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy"
Jul 28 20:14:45.791383 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.791356    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:45.794188 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.794167    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zkwl\""
Jul 28 20:14:45.798955 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.798925    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"]
Jul 28 20:14:45.865398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.865350    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caffa31-fd5b-422a-b46f-84c1e875ea0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7\" (UID: \"1caffa31-fd5b-422a-b46f-84c1e875ea0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:45.966743 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.966696    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caffa31-fd5b-422a-b46f-84c1e875ea0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7\" (UID: \"1caffa31-fd5b-422a-b46f-84c1e875ea0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:45.967396 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:45.967378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caffa31-fd5b-422a-b46f-84c1e875ea0b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7\" (UID: \"1caffa31-fd5b-422a-b46f-84c1e875ea0b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:46.104110 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:46.104012    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:46.266104 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:46.266073    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"]
Jul 28 20:14:46.267823 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:46.267792    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caffa31_fd5b_422a_b46f_84c1e875ea0b.slice/crio-9376fb7602b57cdf11625efe174f720c6b7c337a58fc9574fd83ff9d5b5cb6bd WatchSource:0}: Error finding container 9376fb7602b57cdf11625efe174f720c6b7c337a58fc9574fd83ff9d5b5cb6bd: Status 404 returned error can't find the container with id 9376fb7602b57cdf11625efe174f720c6b7c337a58fc9574fd83ff9d5b5cb6bd
Jul 28 20:14:46.951443 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:46.951408    2578 generic.go:356] "Generic (PLEG): container finished" podID="1caffa31-fd5b-422a-b46f-84c1e875ea0b" containerID="af2105624b243c2499c6260d882b5275c7bde2026d91b9a980cdae68bceba451" exitCode=0
Jul 28 20:14:46.951834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:46.951492    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7" event={"ID":"1caffa31-fd5b-422a-b46f-84c1e875ea0b","Type":"ContainerDied","Data":"af2105624b243c2499c6260d882b5275c7bde2026d91b9a980cdae68bceba451"}
Jul 28 20:14:46.951834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:46.951545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7" event={"ID":"1caffa31-fd5b-422a-b46f-84c1e875ea0b","Type":"ContainerStarted","Data":"9376fb7602b57cdf11625efe174f720c6b7c337a58fc9574fd83ff9d5b5cb6bd"}
Jul 28 20:14:48.101843 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.101814    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:48.185408 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.185363    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caffa31-fd5b-422a-b46f-84c1e875ea0b-maas-api-service-ca\") pod \"1caffa31-fd5b-422a-b46f-84c1e875ea0b\" (UID: \"1caffa31-fd5b-422a-b46f-84c1e875ea0b\") "
Jul 28 20:14:48.185728 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.185704    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1caffa31-fd5b-422a-b46f-84c1e875ea0b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1caffa31-fd5b-422a-b46f-84c1e875ea0b" (UID: "1caffa31-fd5b-422a-b46f-84c1e875ea0b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:48.286936 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.286896    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1caffa31-fd5b-422a-b46f-84c1e875ea0b-maas-api-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:48.963424 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.963390    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"
Jul 28 20:14:48.963602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.963443    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7" event={"ID":"1caffa31-fd5b-422a-b46f-84c1e875ea0b","Type":"ContainerDied","Data":"9376fb7602b57cdf11625efe174f720c6b7c337a58fc9574fd83ff9d5b5cb6bd"}
Jul 28 20:14:48.963602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:48.963493    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9376fb7602b57cdf11625efe174f720c6b7c337a58fc9574fd83ff9d5b5cb6bd"
Jul 28 20:14:51.473318 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.473279    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"]
Jul 28 20:14:51.473768 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.473561    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5" podUID="63c4adbe-1170-49ce-a323-cc88dd62f49c" containerName="maas-api" containerID="cri-o://97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b" gracePeriod=30
Jul 28 20:14:51.527169 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.527131    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"]
Jul 28 20:14:51.527452 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.527423    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b" podUID="9bc2d155-b57f-4279-8c8d-7186c94147d4" containerName="payload-processing" containerID="cri-o://aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271" gracePeriod=30
Jul 28 20:14:51.533559 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.533527    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"]
Jul 28 20:14:51.533851 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.533763    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7" podUID="94f2d7a3-57f4-4fa5-8229-0dada07ec963" containerName="payload-pre-processing" containerID="cri-o://d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e" gracePeriod=30
Jul 28 20:14:51.663877 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.663843    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"]
Jul 28 20:14:51.665853 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.665823    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-916f7ae5-pxjx7"]
Jul 28 20:14:51.807717 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.807689    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:51.842873 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.842834    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:51.855169 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.855146    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:51.919948 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.919903    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4adbe-1170-49ce-a323-cc88dd62f49c-openshift-service-ca\") pod \"63c4adbe-1170-49ce-a323-cc88dd62f49c\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") "
Jul 28 20:14:51.919948 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.919952    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p72zf\" (UniqueName: \"kubernetes.io/projected/9bc2d155-b57f-4279-8c8d-7186c94147d4-kube-api-access-p72zf\") pod \"9bc2d155-b57f-4279-8c8d-7186c94147d4\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") "
Jul 28 20:14:51.920211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920004    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f2d7a3-57f4-4fa5-8229-0dada07ec963-plugins-config-volume\") pod \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") "
Jul 28 20:14:51.920211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920055    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25gjp\" (UniqueName: \"kubernetes.io/projected/63c4adbe-1170-49ce-a323-cc88dd62f49c-kube-api-access-25gjp\") pod \"63c4adbe-1170-49ce-a323-cc88dd62f49c\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") "
Jul 28 20:14:51.920211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920111    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc2d155-b57f-4279-8c8d-7186c94147d4-plugins-config-volume\") pod \"9bc2d155-b57f-4279-8c8d-7186c94147d4\" (UID: \"9bc2d155-b57f-4279-8c8d-7186c94147d4\") "
Jul 28 20:14:51.920211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4adbe-1170-49ce-a323-cc88dd62f49c-maas-api-tls\") pod \"63c4adbe-1170-49ce-a323-cc88dd62f49c\" (UID: \"63c4adbe-1170-49ce-a323-cc88dd62f49c\") "
Jul 28 20:14:51.920211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67gwl\" (UniqueName: \"kubernetes.io/projected/94f2d7a3-57f4-4fa5-8229-0dada07ec963-kube-api-access-67gwl\") pod \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\" (UID: \"94f2d7a3-57f4-4fa5-8229-0dada07ec963\") "
Jul 28 20:14:51.920501 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920353    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c4adbe-1170-49ce-a323-cc88dd62f49c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "63c4adbe-1170-49ce-a323-cc88dd62f49c" (UID: "63c4adbe-1170-49ce-a323-cc88dd62f49c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:51.920566 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920537    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/63c4adbe-1170-49ce-a323-cc88dd62f49c-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:51.920624 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f2d7a3-57f4-4fa5-8229-0dada07ec963-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "94f2d7a3-57f4-4fa5-8229-0dada07ec963" (UID: "94f2d7a3-57f4-4fa5-8229-0dada07ec963"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:51.920735 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.920701    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bc2d155-b57f-4279-8c8d-7186c94147d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9bc2d155-b57f-4279-8c8d-7186c94147d4" (UID: "9bc2d155-b57f-4279-8c8d-7186c94147d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:14:51.922805 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.922777    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bc2d155-b57f-4279-8c8d-7186c94147d4-kube-api-access-p72zf" (OuterVolumeSpecName: "kube-api-access-p72zf") pod "9bc2d155-b57f-4279-8c8d-7186c94147d4" (UID: "9bc2d155-b57f-4279-8c8d-7186c94147d4"). InnerVolumeSpecName "kube-api-access-p72zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:51.923176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.923143    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c4adbe-1170-49ce-a323-cc88dd62f49c-kube-api-access-25gjp" (OuterVolumeSpecName: "kube-api-access-25gjp") pod "63c4adbe-1170-49ce-a323-cc88dd62f49c" (UID: "63c4adbe-1170-49ce-a323-cc88dd62f49c"). InnerVolumeSpecName "kube-api-access-25gjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:51.923325 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.923299    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f2d7a3-57f4-4fa5-8229-0dada07ec963-kube-api-access-67gwl" (OuterVolumeSpecName: "kube-api-access-67gwl") pod "94f2d7a3-57f4-4fa5-8229-0dada07ec963" (UID: "94f2d7a3-57f4-4fa5-8229-0dada07ec963"). InnerVolumeSpecName "kube-api-access-67gwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:14:51.923408 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.923381    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c4adbe-1170-49ce-a323-cc88dd62f49c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "63c4adbe-1170-49ce-a323-cc88dd62f49c" (UID: "63c4adbe-1170-49ce-a323-cc88dd62f49c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:14:51.982276 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.982228    2578 generic.go:356] "Generic (PLEG): container finished" podID="9bc2d155-b57f-4279-8c8d-7186c94147d4" containerID="aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271" exitCode=0
Jul 28 20:14:51.982453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.982301    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"
Jul 28 20:14:51.982453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.982312    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b" event={"ID":"9bc2d155-b57f-4279-8c8d-7186c94147d4","Type":"ContainerDied","Data":"aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271"}
Jul 28 20:14:51.982453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.982400    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b" event={"ID":"9bc2d155-b57f-4279-8c8d-7186c94147d4","Type":"ContainerDied","Data":"e9c055a38349ea355add5e2d202c69bc749ac6b450775e3eea8f21b992f57253"}
Jul 28 20:14:51.982453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.982430    2578 scope.go:117] "RemoveContainer" containerID="aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271"
Jul 28 20:14:51.983723 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.983698    2578 generic.go:356] "Generic (PLEG): container finished" podID="94f2d7a3-57f4-4fa5-8229-0dada07ec963" containerID="d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e" exitCode=0
Jul 28 20:14:51.983827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.983742    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"
Jul 28 20:14:51.983827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.983783    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7" event={"ID":"94f2d7a3-57f4-4fa5-8229-0dada07ec963","Type":"ContainerDied","Data":"d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e"}
Jul 28 20:14:51.983949 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.983859    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7" event={"ID":"94f2d7a3-57f4-4fa5-8229-0dada07ec963","Type":"ContainerDied","Data":"f839a76bd852c0f1d18f57f030a6837e8cf745994d1fefa1d6b7e0d66126ca50"}
Jul 28 20:14:51.985520 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.985495    2578 generic.go:356] "Generic (PLEG): container finished" podID="63c4adbe-1170-49ce-a323-cc88dd62f49c" containerID="97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b" exitCode=0
Jul 28 20:14:51.985637 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.985560    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5" event={"ID":"63c4adbe-1170-49ce-a323-cc88dd62f49c","Type":"ContainerDied","Data":"97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b"}
Jul 28 20:14:51.985637 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.985565    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"
Jul 28 20:14:51.985637 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.985602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5" event={"ID":"63c4adbe-1170-49ce-a323-cc88dd62f49c","Type":"ContainerDied","Data":"5ecfe5ef144d8dfa45ed4352c11daf868a174ab0635e023955f27bba413bf034"}
Jul 28 20:14:51.993979 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.993961    2578 scope.go:117] "RemoveContainer" containerID="aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271"
Jul 28 20:14:51.994316 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:51.994288    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271\": container with ID starting with aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271 not found: ID does not exist" containerID="aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271"
Jul 28 20:14:51.994413 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.994323    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271"} err="failed to get container status \"aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271\": rpc error: code = NotFound desc = could not find container \"aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271\": container with ID starting with aedcb4a3a7df37ad7f5a23df810bdef8485b2f639aea700ced8ea8364d3a6271 not found: ID does not exist"
Jul 28 20:14:51.994413 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:51.994342    2578 scope.go:117] "RemoveContainer" containerID="d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e"
Jul 28 20:14:52.002904 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.002887    2578 scope.go:117] "RemoveContainer" containerID="d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e"
Jul 28 20:14:52.003171 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:52.003152    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e\": container with ID starting with d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e not found: ID does not exist" containerID="d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e"
Jul 28 20:14:52.003244 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.003178    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e"} err="failed to get container status \"d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e\": rpc error: code = NotFound desc = could not find container \"d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e\": container with ID starting with d31c794f159febabe9f152dc01cf78b48c672c61fa6fc0efbc99c6cc6e1fd09e not found: ID does not exist"
Jul 28 20:14:52.003244 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.003196    2578 scope.go:117] "RemoveContainer" containerID="97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b"
Jul 28 20:14:52.012566 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.012539    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"]
Jul 28 20:14:52.013646 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.013625    2578 scope.go:117] "RemoveContainer" containerID="97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b"
Jul 28 20:14:52.013929 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:52.013908    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b\": container with ID starting with 97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b not found: ID does not exist" containerID="97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b"
Jul 28 20:14:52.014000 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.013936    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b"} err="failed to get container status \"97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b\": rpc error: code = NotFound desc = could not find container \"97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b\": container with ID starting with 97ea11f2c785093d9f93da5137bbbadaceca2af626a5e6207eeff115d6c66f3b not found: ID does not exist"
Jul 28 20:14:52.015061 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.015039    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-916f7ae5-97597d8fc-lv69b"]
Jul 28 20:14:52.021830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.021804    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bc2d155-b57f-4279-8c8d-7186c94147d4-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:52.021830 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.021832    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/63c4adbe-1170-49ce-a323-cc88dd62f49c-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:52.021987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.021842    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67gwl\" (UniqueName: \"kubernetes.io/projected/94f2d7a3-57f4-4fa5-8229-0dada07ec963-kube-api-access-67gwl\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:52.021987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.021851    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p72zf\" (UniqueName: \"kubernetes.io/projected/9bc2d155-b57f-4279-8c8d-7186c94147d4-kube-api-access-p72zf\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:52.021987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.021860    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/94f2d7a3-57f4-4fa5-8229-0dada07ec963-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:52.021987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.021869    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25gjp\" (UniqueName: \"kubernetes.io/projected/63c4adbe-1170-49ce-a323-cc88dd62f49c-kube-api-access-25gjp\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:14:52.024387 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.024361    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"]
Jul 28 20:14:52.026211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.026190    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-916f7ae5-7b75645485-hwqt5"]
Jul 28 20:14:52.036930 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.036900    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"]
Jul 28 20:14:52.040233 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.040206    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-916f7ae5-68bcf4cfc6-thlq7"]
Jul 28 20:14:52.572736 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.572699    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"]
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573003    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bc2d155-b57f-4279-8c8d-7186c94147d4" containerName="payload-processing"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573015    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bc2d155-b57f-4279-8c8d-7186c94147d4" containerName="payload-processing"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573024    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63c4adbe-1170-49ce-a323-cc88dd62f49c" containerName="maas-api"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573030    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="63c4adbe-1170-49ce-a323-cc88dd62f49c" containerName="maas-api"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573040    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caffa31-fd5b-422a-b46f-84c1e875ea0b" containerName="revoke-keys"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573046    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caffa31-fd5b-422a-b46f-84c1e875ea0b" containerName="revoke-keys"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573056    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94f2d7a3-57f4-4fa5-8229-0dada07ec963" containerName="payload-pre-processing"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573061    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="94f2d7a3-57f4-4fa5-8229-0dada07ec963" containerName="payload-pre-processing"
Jul 28 20:14:52.573111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573115    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="94f2d7a3-57f4-4fa5-8229-0dada07ec963" containerName="payload-pre-processing"
Jul 28 20:14:52.573572 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573125    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1caffa31-fd5b-422a-b46f-84c1e875ea0b" containerName="revoke-keys"
Jul 28 20:14:52.573572 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573131    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="63c4adbe-1170-49ce-a323-cc88dd62f49c" containerName="maas-api"
Jul 28 20:14:52.573572 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573138    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bc2d155-b57f-4279-8c8d-7186c94147d4" containerName="payload-processing"
Jul 28 20:14:52.573572 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.573144    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d94e35c5-4410-4953-8133-bc687a23ce97" containerName="istio-proxy"
Jul 28 20:14:52.577729 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.577706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.580263 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.580238    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-edc4df58-openshift-default-dockercfg-zlfp6\""
Jul 28 20:14:52.589230 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.589200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"]
Jul 28 20:14:52.727536 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727500    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-certs\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.727536 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727537    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-data\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.727762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727675    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-socket\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.727762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-credential-socket\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.727762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727742    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b380c359-3d9a-4ef3-a2c8-6eae45024292-istiod-ca-cert\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.727891 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6pr6\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-kube-api-access-q6pr6\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.727891 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-token\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.728072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-podinfo\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.728072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.727980    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-envoy\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.828959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.828903    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6pr6\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-kube-api-access-q6pr6\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.828959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.828967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-token\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.828994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-podinfo\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829027    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-envoy\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-certs\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829068    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-data\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829097    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-socket\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-credential-socket\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829155    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b380c359-3d9a-4ef3-a2c8-6eae45024292-istiod-ca-cert\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829583 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829554    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-certs\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829670 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829642    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-socket\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829732 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829685    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-data\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829719    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-credential-socket\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.829881 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.829864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b380c359-3d9a-4ef3-a2c8-6eae45024292-istiod-ca-cert\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.831558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.831537    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-envoy\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.831979 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.831962    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-podinfo\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.838016 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.837989    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-token\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.838120 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.838057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6pr6\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-kube-api-access-q6pr6\") pod \"e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") " pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:52.889163 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:52.889123    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:53.047012 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.046985    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"]
Jul 28 20:14:53.048976 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:53.048941    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb380c359_3d9a_4ef3_a2c8_6eae45024292.slice/crio-0e4bb22a792051d3525405aecd7964380f312aaeab0e44d19111549e327dd671 WatchSource:0}: Error finding container 0e4bb22a792051d3525405aecd7964380f312aaeab0e44d19111549e327dd671: Status 404 returned error can't find the container with id 0e4bb22a792051d3525405aecd7964380f312aaeab0e44d19111549e327dd671
Jul 28 20:14:53.051091 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.051060    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:14:53.051173 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.051125    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:14:53.051173 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.051158    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:14:53.651458 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.651408    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1caffa31-fd5b-422a-b46f-84c1e875ea0b" path="/var/lib/kubelet/pods/1caffa31-fd5b-422a-b46f-84c1e875ea0b/volumes"
Jul 28 20:14:53.651989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.651965    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63c4adbe-1170-49ce-a323-cc88dd62f49c" path="/var/lib/kubelet/pods/63c4adbe-1170-49ce-a323-cc88dd62f49c/volumes"
Jul 28 20:14:53.652402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.652383    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94f2d7a3-57f4-4fa5-8229-0dada07ec963" path="/var/lib/kubelet/pods/94f2d7a3-57f4-4fa5-8229-0dada07ec963/volumes"
Jul 28 20:14:53.652754 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:53.652738    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bc2d155-b57f-4279-8c8d-7186c94147d4" path="/var/lib/kubelet/pods/9bc2d155-b57f-4279-8c8d-7186c94147d4/volumes"
Jul 28 20:14:54.004199 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:54.004104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" event={"ID":"b380c359-3d9a-4ef3-a2c8-6eae45024292","Type":"ContainerStarted","Data":"fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"}
Jul 28 20:14:54.004199 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:54.004159    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" event={"ID":"b380c359-3d9a-4ef3-a2c8-6eae45024292","Type":"ContainerStarted","Data":"0e4bb22a792051d3525405aecd7964380f312aaeab0e44d19111549e327dd671"}
Jul 28 20:14:54.026827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:54.026776    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podStartSLOduration=2.026758389 podStartE2EDuration="2.026758389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:54.025260861 +0000 UTC m=+1696.925477716" watchObservedRunningTime="2026-07-28 20:14:54.026758389 +0000 UTC m=+1696.926975230"
Jul 28 20:14:54.889628 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:54.889588    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:54.894905 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:54.894880    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:55.015496 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.015467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:55.016613 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.016594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:14:55.480176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.480141    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"]
Jul 28 20:14:55.486507 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.486479    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.490617 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.490590    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:14:55.490798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.490590    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:14:55.490798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.490590    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-edc4df58-serving-cert\""
Jul 28 20:14:55.490798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.490598    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:14:55.495964 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.495936    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"]
Jul 28 20:14:55.551607 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.551563    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.551794 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.551638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpq2b\" (UniqueName: \"kubernetes.io/projected/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-kube-api-access-vpq2b\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.551794 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.551697    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-openshift-service-ca\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.652374 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.652344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpq2b\" (UniqueName: \"kubernetes.io/projected/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-kube-api-access-vpq2b\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.652560 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.652382    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-openshift-service-ca\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.652560 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.652425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.652560 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:55.652519    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-serving-cert: secret "maas-api-e2e-ait-edc4df58-serving-cert" not found
Jul 28 20:14:55.652672 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:14:55.652587    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls podName:e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055 nodeName:}" failed. No retries permitted until 2026-07-28 20:14:56.152567126 +0000 UTC m=+1699.052783946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls") pod "maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" (UID: "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055") : secret "maas-api-e2e-ait-edc4df58-serving-cert" not found
Jul 28 20:14:55.653078 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.653057    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-openshift-service-ca\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.662161 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.662129    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpq2b\" (UniqueName: \"kubernetes.io/projected/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-kube-api-access-vpq2b\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:55.773213 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:55.773181    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:14:56.156774 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.156660    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:56.159452 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.159426    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls\") pod \"maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:56.257157 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.257111    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"]
Jul 28 20:14:56.261285 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.261251    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.264102 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.264071    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-edc4df58-dockercfg-bmjwm\""
Jul 28 20:14:56.264249 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.264181    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-edc4df58\""
Jul 28 20:14:56.274821 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.274789    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"]
Jul 28 20:14:56.296284 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.296243    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"]
Jul 28 20:14:56.300210 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.300189    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.309043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.309016    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"]
Jul 28 20:14:56.358050 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.358015    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pgw\" (UniqueName: \"kubernetes.io/projected/28618b7b-fa02-4e90-a94c-9403907655a6-kube-api-access-b2pgw\") pod \"payload-processing-e2e-ait-edc4df58-69749b846-jtgkf\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") " pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.358218 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.358061    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28618b7b-fa02-4e90-a94c-9403907655a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc4df58-69749b846-jtgkf\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") " pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.358218 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.358085    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6635b18b-08c4-4609-8920-2de67ee45e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.358218 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.358173    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-797f9\" (UniqueName: \"kubernetes.io/projected/6635b18b-08c4-4609-8920-2de67ee45e85-kube-api-access-797f9\") pod \"payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.398687 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.398644    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:56.459030 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.458982    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6635b18b-08c4-4609-8920-2de67ee45e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.459196 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.459077    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-797f9\" (UniqueName: \"kubernetes.io/projected/6635b18b-08c4-4609-8920-2de67ee45e85-kube-api-access-797f9\") pod \"payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.459196 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.459144    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pgw\" (UniqueName: \"kubernetes.io/projected/28618b7b-fa02-4e90-a94c-9403907655a6-kube-api-access-b2pgw\") pod \"payload-processing-e2e-ait-edc4df58-69749b846-jtgkf\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") " pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.459196 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.459181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28618b7b-fa02-4e90-a94c-9403907655a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc4df58-69749b846-jtgkf\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") " pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.459993 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.459963    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6635b18b-08c4-4609-8920-2de67ee45e85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.460106 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.460056    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28618b7b-fa02-4e90-a94c-9403907655a6-plugins-config-volume\") pod \"payload-processing-e2e-ait-edc4df58-69749b846-jtgkf\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") " pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.469859 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.469825    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-797f9\" (UniqueName: \"kubernetes.io/projected/6635b18b-08c4-4609-8920-2de67ee45e85-kube-api-access-797f9\") pod \"payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.469859 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.469856    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pgw\" (UniqueName: \"kubernetes.io/projected/28618b7b-fa02-4e90-a94c-9403907655a6-kube-api-access-b2pgw\") pod \"payload-processing-e2e-ait-edc4df58-69749b846-jtgkf\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") " pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.540844 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.540817    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"]
Jul 28 20:14:56.542712 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:56.542682    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b1d4f0_99b4_4c6b_ad8d_c782ccbf4055.slice/crio-0b14ec1d520c75c0f9ac9648227a90599f67e7578f2cf9e6aff91d2e597c2f87 WatchSource:0}: Error finding container 0b14ec1d520c75c0f9ac9648227a90599f67e7578f2cf9e6aff91d2e597c2f87: Status 404 returned error can't find the container with id 0b14ec1d520c75c0f9ac9648227a90599f67e7578f2cf9e6aff91d2e597c2f87
Jul 28 20:14:56.572841 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.572804    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:56.611684 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.611650    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:14:56.727997 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.727818    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"]
Jul 28 20:14:56.732321 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:56.731689    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6635b18b_08c4_4609_8920_2de67ee45e85.slice/crio-617fe8a8fdd1a58c60e508b2ed8cb40682b06b710d75b4bad6572f15e0a8caea WatchSource:0}: Error finding container 617fe8a8fdd1a58c60e508b2ed8cb40682b06b710d75b4bad6572f15e0a8caea: Status 404 returned error can't find the container with id 617fe8a8fdd1a58c60e508b2ed8cb40682b06b710d75b4bad6572f15e0a8caea
Jul 28 20:14:56.777622 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:56.777572    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"]
Jul 28 20:14:56.780368 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:14:56.780342    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28618b7b_fa02_4e90_a94c_9403907655a6.slice/crio-fe63dd82332e4018ab0741862881d59b3003a4864fe2f42dd85c71fda9c11787 WatchSource:0}: Error finding container fe63dd82332e4018ab0741862881d59b3003a4864fe2f42dd85c71fda9c11787: Status 404 returned error can't find the container with id fe63dd82332e4018ab0741862881d59b3003a4864fe2f42dd85c71fda9c11787
Jul 28 20:14:57.030043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.030003    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" event={"ID":"6635b18b-08c4-4609-8920-2de67ee45e85","Type":"ContainerStarted","Data":"a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4"}
Jul 28 20:14:57.030253 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.030071    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" event={"ID":"6635b18b-08c4-4609-8920-2de67ee45e85","Type":"ContainerStarted","Data":"617fe8a8fdd1a58c60e508b2ed8cb40682b06b710d75b4bad6572f15e0a8caea"}
Jul 28 20:14:57.030253 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.030100    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:14:57.031743 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.031716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" event={"ID":"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055","Type":"ContainerStarted","Data":"74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea"}
Jul 28 20:14:57.031883 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.031767    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" event={"ID":"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055","Type":"ContainerStarted","Data":"0b14ec1d520c75c0f9ac9648227a90599f67e7578f2cf9e6aff91d2e597c2f87"}
Jul 28 20:14:57.031883 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.031799    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:14:57.033392 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.033365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf" event={"ID":"28618b7b-fa02-4e90-a94c-9403907655a6","Type":"ContainerStarted","Data":"b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66"}
Jul 28 20:14:57.033484 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.033415    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf" event={"ID":"28618b7b-fa02-4e90-a94c-9403907655a6","Type":"ContainerStarted","Data":"fe63dd82332e4018ab0741862881d59b3003a4864fe2f42dd85c71fda9c11787"}
Jul 28 20:14:57.049636 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.049569    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" podStartSLOduration=1.049548154 podStartE2EDuration="1.049548154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:57.049375923 +0000 UTC m=+1699.949592768" watchObservedRunningTime="2026-07-28 20:14:57.049548154 +0000 UTC m=+1699.949764997"
Jul 28 20:14:57.068659 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.068603    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" podStartSLOduration=1.6658776789999998 podStartE2EDuration="2.068584868s" totalImagesPullingTime="402.707189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:57.066064256 +0000 UTC m=+1699.966281099" watchObservedRunningTime="2026-07-28 20:14:57.068584868 +0000 UTC m=+1699.968801709"
Jul 28 20:14:57.081654 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:57.081593    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf" podStartSLOduration=1.081575677 podStartE2EDuration="1.081575677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:14:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:14:57.080677234 +0000 UTC m=+1699.980894074" watchObservedRunningTime="2026-07-28 20:14:57.081575677 +0000 UTC m=+1699.981792518"
Jul 28 20:14:58.038980 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:14:58.038945    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:15:00.128982 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:00.128901    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"]
Jul 28 20:15:00.132584 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:00.132563    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"
Jul 28 20:15:00.135260 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:00.135239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zkwl\""
Jul 28 20:15:00.142719 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:00.142690    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"]
Jul 28 20:15:00.144186 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:00.144168    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"
Jul 28 20:15:00.285586 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:00.285560    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"]
Jul 28 20:15:00.287318 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:15:00.287291    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda625c4ac_004d_42f3_9cbc_384d66aaa1d4.slice/crio-e201bc1ffe18e8a08337693dcb13d6003d0e21591c257a6314744e4750192a85 WatchSource:0}: Error finding container e201bc1ffe18e8a08337693dcb13d6003d0e21591c257a6314744e4750192a85: Status 404 returned error can't find the container with id e201bc1ffe18e8a08337693dcb13d6003d0e21591c257a6314744e4750192a85
Jul 28 20:15:01.055475 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:01.055440    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb" event={"ID":"a625c4ac-004d-42f3-9cbc-384d66aaa1d4","Type":"ContainerStarted","Data":"dd9fbbedf654d70687b3c9cfc5039eba805e3c119247d5cc3124f76c808d3aa6"}
Jul 28 20:15:01.055648 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:01.055494    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb" event={"ID":"a625c4ac-004d-42f3-9cbc-384d66aaa1d4","Type":"ContainerStarted","Data":"e201bc1ffe18e8a08337693dcb13d6003d0e21591c257a6314744e4750192a85"}
Jul 28 20:15:01.071798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:01.071744    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb" podStartSLOduration=1.071731273 podStartE2EDuration="1.071731273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:01.070571033 +0000 UTC m=+1703.970787872" watchObservedRunningTime="2026-07-28 20:15:01.071731273 +0000 UTC m=+1703.971948093"
Jul 28 20:15:02.062223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:02.062187    2578 generic.go:356] "Generic (PLEG): container finished" podID="a625c4ac-004d-42f3-9cbc-384d66aaa1d4" containerID="dd9fbbedf654d70687b3c9cfc5039eba805e3c119247d5cc3124f76c808d3aa6" exitCode=0
Jul 28 20:15:02.062720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:02.062262    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb" event={"ID":"a625c4ac-004d-42f3-9cbc-384d66aaa1d4","Type":"ContainerDied","Data":"dd9fbbedf654d70687b3c9cfc5039eba805e3c119247d5cc3124f76c808d3aa6"}
Jul 28 20:15:03.044685 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:03.044654    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:15:03.238562 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:03.238519    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"
Jul 28 20:15:04.073785 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:04.073749    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb"
Jul 28 20:15:04.073785 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:04.073778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754495-pvbkb" event={"ID":"a625c4ac-004d-42f3-9cbc-384d66aaa1d4","Type":"ContainerDied","Data":"e201bc1ffe18e8a08337693dcb13d6003d0e21591c257a6314744e4750192a85"}
Jul 28 20:15:04.074039 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:04.073825    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e201bc1ffe18e8a08337693dcb13d6003d0e21591c257a6314744e4750192a85"
Jul 28 20:15:06.274892 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:06.274845    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:15:08.040881 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:08.040832    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" podUID="6635b18b-08c4-4609-8920-2de67ee45e85" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.49:9004: connect: connection refused"
Jul 28 20:15:09.045692 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:09.045660    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:15:10.017799 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.017761    2578 patch_prober.go:28] interesting pod/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 28 20:15:10.018014 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.017818    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 28 20:15:10.111187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.111156    2578 generic.go:356] "Generic (PLEG): container finished" podID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerID="fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d" exitCode=0
Jul 28 20:15:10.111612 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.111209    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" event={"ID":"b380c359-3d9a-4ef3-a2c8-6eae45024292","Type":"ContainerDied","Data":"fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"}
Jul 28 20:15:10.111707 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.111690    2578 scope.go:117] "RemoveContainer" containerID="fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"
Jul 28 20:15:10.112782 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.112752    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:10.112897 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.112837    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:10.112897 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:10.112886    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:11.131684 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:11.131623    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" event={"ID":"b380c359-3d9a-4ef3-a2c8-6eae45024292","Type":"ContainerStarted","Data":"14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba"}
Jul 28 20:15:11.889341 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:11.889301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:15:11.889588 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:11.889570    2578 patch_prober.go:28] interesting pod/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 28 20:15:11.889636 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:11.889613    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 28 20:15:12.889974 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:12.889939    2578 patch_prober.go:28] interesting pod/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 28 20:15:12.890407 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:12.889997    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 28 20:15:13.889968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:13.889925    2578 patch_prober.go:28] interesting pod/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 28 20:15:13.890166 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:13.890001    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 28 20:15:14.890398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:14.890352    2578 patch_prober.go:28] interesting pod/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body=
Jul 28 20:15:14.890837 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:14.890421    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused"
Jul 28 20:15:15.895062 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:15.895023    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:15:15.895613 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:15.895573    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:15:15.896396 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:15.896375    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:15:18.039471 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:18.039443    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:15:26.279243 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.279205    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"]
Jul 28 20:15:26.279751 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.279433    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" podUID="e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" containerName="maas-api" containerID="cri-o://74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea" gracePeriod=30
Jul 28 20:15:26.368614 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.368568    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"]
Jul 28 20:15:26.369135 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.368947    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf" podUID="28618b7b-fa02-4e90-a94c-9403907655a6" containerName="payload-processing" containerID="cri-o://b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66" gracePeriod=30
Jul 28 20:15:26.372188 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.372159    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"]
Jul 28 20:15:26.372451 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.372417    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" podUID="6635b18b-08c4-4609-8920-2de67ee45e85" containerName="payload-pre-processing" containerID="cri-o://a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4" gracePeriod=30
Jul 28 20:15:26.622823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.622791    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:15:26.689705 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.689677    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:15:26.693865 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.693833    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:15:26.749669 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749634    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-openshift-service-ca\") pod \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") "
Jul 28 20:15:26.749863 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749723    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpq2b\" (UniqueName: \"kubernetes.io/projected/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-kube-api-access-vpq2b\") pod \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") "
Jul 28 20:15:26.749863 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28618b7b-fa02-4e90-a94c-9403907655a6-plugins-config-volume\") pod \"28618b7b-fa02-4e90-a94c-9403907655a6\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") "
Jul 28 20:15:26.749863 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749772    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6635b18b-08c4-4609-8920-2de67ee45e85-plugins-config-volume\") pod \"6635b18b-08c4-4609-8920-2de67ee45e85\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") "
Jul 28 20:15:26.749863 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749797    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2pgw\" (UniqueName: \"kubernetes.io/projected/28618b7b-fa02-4e90-a94c-9403907655a6-kube-api-access-b2pgw\") pod \"28618b7b-fa02-4e90-a94c-9403907655a6\" (UID: \"28618b7b-fa02-4e90-a94c-9403907655a6\") "
Jul 28 20:15:26.750070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749878    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-797f9\" (UniqueName: \"kubernetes.io/projected/6635b18b-08c4-4609-8920-2de67ee45e85-kube-api-access-797f9\") pod \"6635b18b-08c4-4609-8920-2de67ee45e85\" (UID: \"6635b18b-08c4-4609-8920-2de67ee45e85\") "
Jul 28 20:15:26.750070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.749906    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls\") pod \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\" (UID: \"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055\") "
Jul 28 20:15:26.750070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.750049    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" (UID: "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:15:26.750224 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.750149    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6635b18b-08c4-4609-8920-2de67ee45e85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6635b18b-08c4-4609-8920-2de67ee45e85" (UID: "6635b18b-08c4-4609-8920-2de67ee45e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:15:26.750224 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.750171    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:26.750224 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.750179    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28618b7b-fa02-4e90-a94c-9403907655a6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "28618b7b-fa02-4e90-a94c-9403907655a6" (UID: "28618b7b-fa02-4e90-a94c-9403907655a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:15:26.752471 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.752416    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28618b7b-fa02-4e90-a94c-9403907655a6-kube-api-access-b2pgw" (OuterVolumeSpecName: "kube-api-access-b2pgw") pod "28618b7b-fa02-4e90-a94c-9403907655a6" (UID: "28618b7b-fa02-4e90-a94c-9403907655a6"). InnerVolumeSpecName "kube-api-access-b2pgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:26.752471 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.752447    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6635b18b-08c4-4609-8920-2de67ee45e85-kube-api-access-797f9" (OuterVolumeSpecName: "kube-api-access-797f9") pod "6635b18b-08c4-4609-8920-2de67ee45e85" (UID: "6635b18b-08c4-4609-8920-2de67ee45e85"). InnerVolumeSpecName "kube-api-access-797f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:26.752779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.752758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" (UID: "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:15:26.752826 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.752759    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-kube-api-access-vpq2b" (OuterVolumeSpecName: "kube-api-access-vpq2b") pod "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" (UID: "e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055"). InnerVolumeSpecName "kube-api-access-vpq2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:26.851052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.850947    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-797f9\" (UniqueName: \"kubernetes.io/projected/6635b18b-08c4-4609-8920-2de67ee45e85-kube-api-access-797f9\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:26.851052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.850984    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:26.851052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.850998    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpq2b\" (UniqueName: \"kubernetes.io/projected/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055-kube-api-access-vpq2b\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:26.851052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.851010    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/28618b7b-fa02-4e90-a94c-9403907655a6-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:26.851052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.851022    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6635b18b-08c4-4609-8920-2de67ee45e85-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:26.851052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:26.851033    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2pgw\" (UniqueName: \"kubernetes.io/projected/28618b7b-fa02-4e90-a94c-9403907655a6-kube-api-access-b2pgw\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:27.221744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.221654    2578 generic.go:356] "Generic (PLEG): container finished" podID="e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" containerID="74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea" exitCode=0
Jul 28 20:15:27.221744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.221697    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"
Jul 28 20:15:27.221981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.221741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" event={"ID":"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055","Type":"ContainerDied","Data":"74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea"}
Jul 28 20:15:27.221981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.221800    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4" event={"ID":"e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055","Type":"ContainerDied","Data":"0b14ec1d520c75c0f9ac9648227a90599f67e7578f2cf9e6aff91d2e597c2f87"}
Jul 28 20:15:27.221981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.221830    2578 scope.go:117] "RemoveContainer" containerID="74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea"
Jul 28 20:15:27.223140 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.223118    2578 generic.go:356] "Generic (PLEG): container finished" podID="28618b7b-fa02-4e90-a94c-9403907655a6" containerID="b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66" exitCode=0
Jul 28 20:15:27.223222 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.223155    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"
Jul 28 20:15:27.223222 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.223200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf" event={"ID":"28618b7b-fa02-4e90-a94c-9403907655a6","Type":"ContainerDied","Data":"b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66"}
Jul 28 20:15:27.223362 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.223289    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf" event={"ID":"28618b7b-fa02-4e90-a94c-9403907655a6","Type":"ContainerDied","Data":"fe63dd82332e4018ab0741862881d59b3003a4864fe2f42dd85c71fda9c11787"}
Jul 28 20:15:27.224748 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.224717    2578 generic.go:356] "Generic (PLEG): container finished" podID="6635b18b-08c4-4609-8920-2de67ee45e85" containerID="a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4" exitCode=0
Jul 28 20:15:27.224875 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.224759    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" event={"ID":"6635b18b-08c4-4609-8920-2de67ee45e85","Type":"ContainerDied","Data":"a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4"}
Jul 28 20:15:27.224875 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.224812    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6" event={"ID":"6635b18b-08c4-4609-8920-2de67ee45e85","Type":"ContainerDied","Data":"617fe8a8fdd1a58c60e508b2ed8cb40682b06b710d75b4bad6572f15e0a8caea"}
Jul 28 20:15:27.224875 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.224769    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"
Jul 28 20:15:27.232322 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.232297    2578 scope.go:117] "RemoveContainer" containerID="74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea"
Jul 28 20:15:27.232627 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:27.232604    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea\": container with ID starting with 74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea not found: ID does not exist" containerID="74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea"
Jul 28 20:15:27.232697 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.232641    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea"} err="failed to get container status \"74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea\": rpc error: code = NotFound desc = could not find container \"74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea\": container with ID starting with 74bb0ae09a0391786a9eb96bba3d8986f157aa026b65dfabd87e2090c607d3ea not found: ID does not exist"
Jul 28 20:15:27.232697 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.232665    2578 scope.go:117] "RemoveContainer" containerID="b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66"
Jul 28 20:15:27.244173 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.244147    2578 scope.go:117] "RemoveContainer" containerID="b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66"
Jul 28 20:15:27.245144 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:27.245117    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66\": container with ID starting with b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66 not found: ID does not exist" containerID="b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66"
Jul 28 20:15:27.245280 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.245151    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66"} err="failed to get container status \"b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66\": rpc error: code = NotFound desc = could not find container \"b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66\": container with ID starting with b9468020a2b295e6b042c2aa60c8175d6e89f8504955d8afe740576ea6587b66 not found: ID does not exist"
Jul 28 20:15:27.245280 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.245170    2578 scope.go:117] "RemoveContainer" containerID="a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4"
Jul 28 20:15:27.252839 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.252812    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"]
Jul 28 20:15:27.255548 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.255526    2578 scope.go:117] "RemoveContainer" containerID="a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4"
Jul 28 20:15:27.255876 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:27.255856    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4\": container with ID starting with a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4 not found: ID does not exist" containerID="a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4"
Jul 28 20:15:27.255915 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.255883    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4"} err="failed to get container status \"a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4\": rpc error: code = NotFound desc = could not find container \"a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4\": container with ID starting with a73fbe520a3ea5a6f6baa8fa6f4bf8f41b5e4879c1d3d68a5a7442554f8301d4 not found: ID does not exist"
Jul 28 20:15:27.256668 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.256648    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-edc4df58-565c98df4c-tcdz4"]
Jul 28 20:15:27.281058 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.281022    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"]
Jul 28 20:15:27.286176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.286146    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-edc4df58-69749b846-jtgkf"]
Jul 28 20:15:27.298708 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.298676    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"]
Jul 28 20:15:27.303373 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.303343    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-edc4df58-57f74ff6cb-jckg6"]
Jul 28 20:15:27.652256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.652221    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28618b7b-fa02-4e90-a94c-9403907655a6" path="/var/lib/kubelet/pods/28618b7b-fa02-4e90-a94c-9403907655a6/volumes"
Jul 28 20:15:27.652610 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.652596    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6635b18b-08c4-4609-8920-2de67ee45e85" path="/var/lib/kubelet/pods/6635b18b-08c4-4609-8920-2de67ee45e85/volumes"
Jul 28 20:15:27.652936 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.652924    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" path="/var/lib/kubelet/pods/e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055/volumes"
Jul 28 20:15:27.812837 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.812799    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"]
Jul 28 20:15:27.813015 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:27.812994    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" containerID="cri-o://14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba" gracePeriod=30
Jul 28 20:15:28.611328 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611288    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"]
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611601    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6635b18b-08c4-4609-8920-2de67ee45e85" containerName="payload-pre-processing"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611611    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="6635b18b-08c4-4609-8920-2de67ee45e85" containerName="payload-pre-processing"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611621    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" containerName="maas-api"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611627    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" containerName="maas-api"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611641    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a625c4ac-004d-42f3-9cbc-384d66aaa1d4" containerName="cleanup"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611646    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="a625c4ac-004d-42f3-9cbc-384d66aaa1d4" containerName="cleanup"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611657    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28618b7b-fa02-4e90-a94c-9403907655a6" containerName="payload-processing"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611662    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="28618b7b-fa02-4e90-a94c-9403907655a6" containerName="payload-processing"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611709    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="28618b7b-fa02-4e90-a94c-9403907655a6" containerName="payload-processing"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611719    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5b1d4f0-99b4-4c6b-ad8d-c782ccbf4055" containerName="maas-api"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611724    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="6635b18b-08c4-4609-8920-2de67ee45e85" containerName="payload-pre-processing"
Jul 28 20:15:28.611746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.611731    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="a625c4ac-004d-42f3-9cbc-384d66aaa1d4" containerName="cleanup"
Jul 28 20:15:28.614754 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.614734    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.617403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.617379    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-10cc1978-openshift-default-dockercfg-8scp2\""
Jul 28 20:15:28.628053 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.628023    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"]
Jul 28 20:15:28.666409 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666367    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7xt\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-kube-api-access-fc7xt\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666420    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-credential-socket\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666470    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-data\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666492    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a30e93-fb6d-4bf4-8183-406306794334-istiod-ca-cert\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666508    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-socket\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666543    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-envoy\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666802 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666568    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a30e93-fb6d-4bf4-8183-406306794334-istio-podinfo\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666802 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666619    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-certs\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.666802 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.666663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-istio-token\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.767667 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.767623    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-certs\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.767848 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.767678    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-istio-token\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.767848 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.767826    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7xt\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-kube-api-access-fc7xt\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.767935 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.767899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-credential-socket\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.767996 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.767979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-data\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768053 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.767996    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-certs\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768108 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768059    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a30e93-fb6d-4bf4-8183-406306794334-istiod-ca-cert\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768166 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768103    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-socket\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768166 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768147    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-envoy\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768313 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768179    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a30e93-fb6d-4bf4-8183-406306794334-istio-podinfo\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768313 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768298    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-data\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768434 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768179    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-credential-socket\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768472 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-socket\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.768989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.768966    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a30e93-fb6d-4bf4-8183-406306794334-istiod-ca-cert\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.770489 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.770468    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-envoy\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.770781 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.770765    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a30e93-fb6d-4bf4-8183-406306794334-istio-podinfo\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.776073 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.776046    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-istio-token\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.776364 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.776343    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7xt\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-kube-api-access-fc7xt\") pod \"e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") " pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:28.925665 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:28.925570    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:29.077240 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.077209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"]
Jul 28 20:15:29.079461 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:15:29.079430    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a30e93_fb6d_4bf4_8183_406306794334.slice/crio-59aa8fe6837a7e7668fac503f7576ded384c125a3f136acb21bb39207d6de46c WatchSource:0}: Error finding container 59aa8fe6837a7e7668fac503f7576ded384c125a3f136acb21bb39207d6de46c: Status 404 returned error can't find the container with id 59aa8fe6837a7e7668fac503f7576ded384c125a3f136acb21bb39207d6de46c
Jul 28 20:15:29.081577 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.081546    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:29.081671 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.081612    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:29.081671 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.081645    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:29.240325 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.240220    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" event={"ID":"23a30e93-fb6d-4bf4-8183-406306794334","Type":"ContainerStarted","Data":"caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"}
Jul 28 20:15:29.240325 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.240318    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" event={"ID":"23a30e93-fb6d-4bf4-8183-406306794334","Type":"ContainerStarted","Data":"59aa8fe6837a7e7668fac503f7576ded384c125a3f136acb21bb39207d6de46c"}
Jul 28 20:15:29.268443 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.268386    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podStartSLOduration=1.268366281 podStartE2EDuration="1.268366281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:29.266036329 +0000 UTC m=+1732.166253171" watchObservedRunningTime="2026-07-28 20:15:29.268366281 +0000 UTC m=+1732.168583124"
Jul 28 20:15:29.925984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.925942    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:29.954202 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:29.954164    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:15:30.761036 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.761001    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"]
Jul 28 20:15:30.763526 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.763508    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:30.766046 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.766017    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-10cc1978-serving-cert\""
Jul 28 20:15:30.766294 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.766255    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:15:30.767373 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.767353    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:15:30.767483 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.767358    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:15:30.775200 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.775174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"]
Jul 28 20:15:30.894783 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.894739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:30.894977 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.894791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4230221e-87ee-4ab2-9a6d-249731205b44-openshift-service-ca\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:30.894977 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.894813    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jhbr\" (UniqueName: \"kubernetes.io/projected/4230221e-87ee-4ab2-9a6d-249731205b44-kube-api-access-8jhbr\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:30.897841 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.897808    2578 patch_prober.go:28] interesting pod/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:15:30.897965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.897865    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:15:30.930824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:30.930792    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:31.005013 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.004920    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.005233 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.005205    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4230221e-87ee-4ab2-9a6d-249731205b44-openshift-service-ca\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.005358 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.005255    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jhbr\" (UniqueName: \"kubernetes.io/projected/4230221e-87ee-4ab2-9a6d-249731205b44-kube-api-access-8jhbr\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.005481 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:31.005462    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-serving-cert: secret "maas-api-e2e-derive-10cc1978-serving-cert" not found
Jul 28 20:15:31.005570 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:31.005551    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls podName:4230221e-87ee-4ab2-9a6d-249731205b44 nodeName:}" failed. No retries permitted until 2026-07-28 20:15:31.505529653 +0000 UTC m=+1734.405746472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls") pod "maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" (UID: "4230221e-87ee-4ab2-9a6d-249731205b44") : secret "maas-api-e2e-derive-10cc1978-serving-cert" not found
Jul 28 20:15:31.006284 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.006242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4230221e-87ee-4ab2-9a6d-249731205b44-openshift-service-ca\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.015358 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.015282    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jhbr\" (UniqueName: \"kubernetes.io/projected/4230221e-87ee-4ab2-9a6d-249731205b44-kube-api-access-8jhbr\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.251357 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.251323    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:31.252325 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.252301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:31.511053 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.511024    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.513728 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.513699    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls\") pod \"maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.642033 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.641999    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"]
Jul 28 20:15:31.644550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.644527    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.647903 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.647874    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-10cc1978\""
Jul 28 20:15:31.648054 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.647940    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-10cc1978-dockercfg-55s4n\""
Jul 28 20:15:31.659365 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.659334    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"]
Jul 28 20:15:31.675052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.675018    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:31.713702 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.713622    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg2d2\" (UniqueName: \"kubernetes.io/projected/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-kube-api-access-tg2d2\") pod \"payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.713908 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.713754    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.814642 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.814601    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg2d2\" (UniqueName: \"kubernetes.io/projected/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-kube-api-access-tg2d2\") pod \"payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.814843 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.814690    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.815479 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.815457    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.826880 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.826857    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"]
Jul 28 20:15:31.828082 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.828058    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg2d2\" (UniqueName: \"kubernetes.io/projected/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-kube-api-access-tg2d2\") pod \"payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:31.829306 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:15:31.829256    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4230221e_87ee_4ab2_9a6d_249731205b44.slice/crio-a0c7c2eba12bebe2a2adbb9f57658e54251fc8c687d5f4a24feacafa6a2688c9 WatchSource:0}: Error finding container a0c7c2eba12bebe2a2adbb9f57658e54251fc8c687d5f4a24feacafa6a2688c9: Status 404 returned error can't find the container with id a0c7c2eba12bebe2a2adbb9f57658e54251fc8c687d5f4a24feacafa6a2688c9
Jul 28 20:15:31.955952 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:31.955909    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:32.092797 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:32.092770    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"]
Jul 28 20:15:32.094875 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:15:32.094847    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc126fcc5_7f33_4239_b4a8_92bf1b3445aa.slice/crio-1d5b7f8e00699cdc035438bd3501da30ee7e38d22b30ba325c37710f4dc56e00 WatchSource:0}: Error finding container 1d5b7f8e00699cdc035438bd3501da30ee7e38d22b30ba325c37710f4dc56e00: Status 404 returned error can't find the container with id 1d5b7f8e00699cdc035438bd3501da30ee7e38d22b30ba325c37710f4dc56e00
Jul 28 20:15:32.257914 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:32.257880    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" event={"ID":"4230221e-87ee-4ab2-9a6d-249731205b44","Type":"ContainerStarted","Data":"a0c7c2eba12bebe2a2adbb9f57658e54251fc8c687d5f4a24feacafa6a2688c9"}
Jul 28 20:15:32.262625 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:32.262581    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb" event={"ID":"c126fcc5-7f33-4239-b4a8-92bf1b3445aa","Type":"ContainerStarted","Data":"800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1"}
Jul 28 20:15:32.262807 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:32.262785    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb" event={"ID":"c126fcc5-7f33-4239-b4a8-92bf1b3445aa","Type":"ContainerStarted","Data":"1d5b7f8e00699cdc035438bd3501da30ee7e38d22b30ba325c37710f4dc56e00"}
Jul 28 20:15:32.262935 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:32.262914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:32.279617 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:32.279573    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb" podStartSLOduration=1.279559391 podStartE2EDuration="1.279559391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:32.276910607 +0000 UTC m=+1735.177127450" watchObservedRunningTime="2026-07-28 20:15:32.279559391 +0000 UTC m=+1735.179776232"
Jul 28 20:15:33.066187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.066156    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:15:33.128820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b380c359-3d9a-4ef3-a2c8-6eae45024292-istiod-ca-cert\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.128820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128771    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6pr6\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-kube-api-access-q6pr6\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.128820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128799    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-podinfo\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.128820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-socket\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.129256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128872    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-certs\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.129256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128891    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-token\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.129256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.128951    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-envoy\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.129256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129016    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-data\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.129256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129043    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-credential-socket\") pod \"b380c359-3d9a-4ef3-a2c8-6eae45024292\" (UID: \"b380c359-3d9a-4ef3-a2c8-6eae45024292\") "
Jul 28 20:15:33.129256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129238    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b380c359-3d9a-4ef3-a2c8-6eae45024292-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:15:33.129602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129410    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b380c359-3d9a-4ef3-a2c8-6eae45024292-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.129602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129597    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:33.129736 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129700    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:33.129910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.129890    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:33.130527 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.130503    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-data" (OuterVolumeSpecName: "istio-data") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:33.131492 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.131455    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:15:33.131730 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.131707    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:15:33.131806 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.131785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-kube-api-access-q6pr6" (OuterVolumeSpecName: "kube-api-access-q6pr6") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "kube-api-access-q6pr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:33.131858 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.131830    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-token" (OuterVolumeSpecName: "istio-token") pod "b380c359-3d9a-4ef3-a2c8-6eae45024292" (UID: "b380c359-3d9a-4ef3-a2c8-6eae45024292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:15:33.230573 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230539    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6pr6\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-kube-api-access-q6pr6\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230573 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230569    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230573 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230580    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230596    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230604    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230612    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230618    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.230824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.230625    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b380c359-3d9a-4ef3-a2c8-6eae45024292-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:15:33.269448 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.269408    2578 generic.go:356] "Generic (PLEG): container finished" podID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerID="14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba" exitCode=0
Jul 28 20:15:33.269625 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.269462    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"
Jul 28 20:15:33.269625 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.269499    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" event={"ID":"b380c359-3d9a-4ef3-a2c8-6eae45024292","Type":"ContainerDied","Data":"14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba"}
Jul 28 20:15:33.269625 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.269559    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf" event={"ID":"b380c359-3d9a-4ef3-a2c8-6eae45024292","Type":"ContainerDied","Data":"0e4bb22a792051d3525405aecd7964380f312aaeab0e44d19111549e327dd671"}
Jul 28 20:15:33.269625 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.269594    2578 scope.go:117] "RemoveContainer" containerID="14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba"
Jul 28 20:15:33.271167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.271142    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" event={"ID":"4230221e-87ee-4ab2-9a6d-249731205b44","Type":"ContainerStarted","Data":"09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2"}
Jul 28 20:15:33.271261 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.271198    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:33.279454 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.279436    2578 scope.go:117] "RemoveContainer" containerID="fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"
Jul 28 20:15:33.287658 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.287638    2578 scope.go:117] "RemoveContainer" containerID="14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba"
Jul 28 20:15:33.287917 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:33.287899    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba\": container with ID starting with 14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba not found: ID does not exist" containerID="14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba"
Jul 28 20:15:33.287965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.287935    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba"} err="failed to get container status \"14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba\": rpc error: code = NotFound desc = could not find container \"14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba\": container with ID starting with 14ed16fb285c480a50b5cd1bace167a182fff01129b95eefe3e2e7d20478c4ba not found: ID does not exist"
Jul 28 20:15:33.287965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.287953    2578 scope.go:117] "RemoveContainer" containerID="fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"
Jul 28 20:15:33.288178 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:15:33.288162    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d\": container with ID starting with fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d not found: ID does not exist" containerID="fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"
Jul 28 20:15:33.288218 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.288184    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d"} err="failed to get container status \"fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d\": rpc error: code = NotFound desc = could not find container \"fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d\": container with ID starting with fa1a509e11ffe4fdd4d7aed3ecd2fd27d8e613d81d03d0ddcb94efdd2c4ce98d not found: ID does not exist"
Jul 28 20:15:33.290565 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.290526    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" podStartSLOduration=2.874789389 podStartE2EDuration="3.29051292s" totalImagesPullingTime="415.723531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:15:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:15:33.289437209 +0000 UTC m=+1736.189654050" watchObservedRunningTime="2026-07-28 20:15:33.29051292 +0000 UTC m=+1736.190729790"
Jul 28 20:15:33.304461 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.304431    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"]
Jul 28 20:15:33.308610 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.308586    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-edc4df58-openshift-default-dcbf466b-xshdf"]
Jul 28 20:15:33.650820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:33.650786    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" path="/var/lib/kubelet/pods/b380c359-3d9a-4ef3-a2c8-6eae45024292/volumes"
Jul 28 20:15:39.283786 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:39.283756    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:15:40.779817 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:40.779778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:15:43.273384 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:43.273350    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:15:45.343301 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:45.343249    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a30e93-fb6d-4bf4-8183-406306794334" containerID="caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47" exitCode=0
Jul 28 20:15:45.343856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:45.343328    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" event={"ID":"23a30e93-fb6d-4bf4-8183-406306794334","Type":"ContainerDied","Data":"caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"}
Jul 28 20:15:45.343856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:45.343833    2578 scope.go:117] "RemoveContainer" containerID="caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"
Jul 28 20:15:45.344906 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:45.344876    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:45.344982 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:45.344965    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:45.345030 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:45.345015    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:15:46.251932 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:46.251887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:46.351008 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:46.350967    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" event={"ID":"23a30e93-fb6d-4bf4-8183-406306794334","Type":"ContainerStarted","Data":"cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e"}
Jul 28 20:15:46.926969 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:46.926928    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:46.927434 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:46.927398    2578 patch_prober.go:28] interesting pod/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 20:15:46.927661 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:46.927631    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 20:15:47.926500 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:47.926459    2578 patch_prober.go:28] interesting pod/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 20:15:47.927179 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:47.927140    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 20:15:48.926300 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:48.926233    2578 patch_prober.go:28] interesting pod/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 20:15:48.926650 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:48.926617    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 20:15:49.926727 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:49.926685    2578 patch_prober.go:28] interesting pod/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 28 20:15:49.927366 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:49.927334    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 28 20:15:50.930596 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:50.930553    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:50.931035 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:50.930840    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:15:50.931520 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:15:50.931501    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:16:01.378424 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.378387    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"]
Jul 28 20:16:01.378966 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.378612    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" podUID="4230221e-87ee-4ab2-9a6d-249731205b44" containerName="maas-api" containerID="cri-o://09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2" gracePeriod=30
Jul 28 20:16:01.442952 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.442908    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"]
Jul 28 20:16:01.443182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.443155    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb" podUID="c126fcc5-7f33-4239-b4a8-92bf1b3445aa" containerName="payload-pre-processing" containerID="cri-o://800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1" gracePeriod=30
Jul 28 20:16:01.707142 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.707108    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:16:01.709089 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.709060    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"]
Jul 28 20:16:01.709323 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.709251    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy" containerID="cri-o://cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e" gracePeriod=30
Jul 28 20:16:01.739742 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.739715    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:16:01.780497 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780463    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4230221e-87ee-4ab2-9a6d-249731205b44-openshift-service-ca\") pod \"4230221e-87ee-4ab2-9a6d-249731205b44\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") "
Jul 28 20:16:01.780705 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780519    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls\") pod \"4230221e-87ee-4ab2-9a6d-249731205b44\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") "
Jul 28 20:16:01.780705 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780548    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jhbr\" (UniqueName: \"kubernetes.io/projected/4230221e-87ee-4ab2-9a6d-249731205b44-kube-api-access-8jhbr\") pod \"4230221e-87ee-4ab2-9a6d-249731205b44\" (UID: \"4230221e-87ee-4ab2-9a6d-249731205b44\") "
Jul 28 20:16:01.780705 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-plugins-config-volume\") pod \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") "
Jul 28 20:16:01.780705 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780606    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg2d2\" (UniqueName: \"kubernetes.io/projected/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-kube-api-access-tg2d2\") pod \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\" (UID: \"c126fcc5-7f33-4239-b4a8-92bf1b3445aa\") "
Jul 28 20:16:01.780957 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780902    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4230221e-87ee-4ab2-9a6d-249731205b44-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4230221e-87ee-4ab2-9a6d-249731205b44" (UID: "4230221e-87ee-4ab2-9a6d-249731205b44"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:16:01.781029 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.780989    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c126fcc5-7f33-4239-b4a8-92bf1b3445aa" (UID: "c126fcc5-7f33-4239-b4a8-92bf1b3445aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:16:01.782916 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.782892    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4230221e-87ee-4ab2-9a6d-249731205b44" (UID: "4230221e-87ee-4ab2-9a6d-249731205b44"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:16:01.782916 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.782886    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-kube-api-access-tg2d2" (OuterVolumeSpecName: "kube-api-access-tg2d2") pod "c126fcc5-7f33-4239-b4a8-92bf1b3445aa" (UID: "c126fcc5-7f33-4239-b4a8-92bf1b3445aa"). InnerVolumeSpecName "kube-api-access-tg2d2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:01.783063 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.782927    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4230221e-87ee-4ab2-9a6d-249731205b44-kube-api-access-8jhbr" (OuterVolumeSpecName: "kube-api-access-8jhbr") pod "4230221e-87ee-4ab2-9a6d-249731205b44" (UID: "4230221e-87ee-4ab2-9a6d-249731205b44"). InnerVolumeSpecName "kube-api-access-8jhbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:01.881703 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.881661    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4230221e-87ee-4ab2-9a6d-249731205b44-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:01.881703 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.881696    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4230221e-87ee-4ab2-9a6d-249731205b44-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:01.881703 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.881707    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jhbr\" (UniqueName: \"kubernetes.io/projected/4230221e-87ee-4ab2-9a6d-249731205b44-kube-api-access-8jhbr\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:01.881957 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.881716    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:01.881957 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:01.881724    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg2d2\" (UniqueName: \"kubernetes.io/projected/c126fcc5-7f33-4239-b4a8-92bf1b3445aa-kube-api-access-tg2d2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:02.435653 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.435613    2578 generic.go:356] "Generic (PLEG): container finished" podID="4230221e-87ee-4ab2-9a6d-249731205b44" containerID="09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2" exitCode=0
Jul 28 20:16:02.436139 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.435666    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"
Jul 28 20:16:02.436139 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.435698    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" event={"ID":"4230221e-87ee-4ab2-9a6d-249731205b44","Type":"ContainerDied","Data":"09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2"}
Jul 28 20:16:02.436139 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.435756    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj" event={"ID":"4230221e-87ee-4ab2-9a6d-249731205b44","Type":"ContainerDied","Data":"a0c7c2eba12bebe2a2adbb9f57658e54251fc8c687d5f4a24feacafa6a2688c9"}
Jul 28 20:16:02.436139 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.435781    2578 scope.go:117] "RemoveContainer" containerID="09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2"
Jul 28 20:16:02.437175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.437153    2578 generic.go:356] "Generic (PLEG): container finished" podID="c126fcc5-7f33-4239-b4a8-92bf1b3445aa" containerID="800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1" exitCode=0
Jul 28 20:16:02.437251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.437211    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"
Jul 28 20:16:02.437405 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.437245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb" event={"ID":"c126fcc5-7f33-4239-b4a8-92bf1b3445aa","Type":"ContainerDied","Data":"800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1"}
Jul 28 20:16:02.437405 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.437322    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb" event={"ID":"c126fcc5-7f33-4239-b4a8-92bf1b3445aa","Type":"ContainerDied","Data":"1d5b7f8e00699cdc035438bd3501da30ee7e38d22b30ba325c37710f4dc56e00"}
Jul 28 20:16:02.445296 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.445259    2578 scope.go:117] "RemoveContainer" containerID="09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2"
Jul 28 20:16:02.445622 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:16:02.445598    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2\": container with ID starting with 09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2 not found: ID does not exist" containerID="09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2"
Jul 28 20:16:02.445719 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.445629    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2"} err="failed to get container status \"09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2\": rpc error: code = NotFound desc = could not find container \"09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2\": container with ID starting with 09de95f67b800c2853c933f05893ecd2574a7a8149dc7316c6835c92ab120bb2 not found: ID does not exist"
Jul 28 20:16:02.445719 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.445646    2578 scope.go:117] "RemoveContainer" containerID="800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1"
Jul 28 20:16:02.454526 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.454505    2578 scope.go:117] "RemoveContainer" containerID="800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1"
Jul 28 20:16:02.454784 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:16:02.454765    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1\": container with ID starting with 800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1 not found: ID does not exist" containerID="800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1"
Jul 28 20:16:02.454839 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.454792    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1"} err="failed to get container status \"800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1\": rpc error: code = NotFound desc = could not find container \"800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1\": container with ID starting with 800a7bd25468340c73672b71f77581edf2e11d46a1a8d081cb72b3c586fb31a1 not found: ID does not exist"
Jul 28 20:16:02.462072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.461988    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"]
Jul 28 20:16:02.464488 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.464465    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-10cc1978-57c5ff8cd5-m5flj"]
Jul 28 20:16:02.482105 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.482070    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"]
Jul 28 20:16:02.486929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:02.486898    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-10cc1978-785cfd648-2kbgb"]
Jul 28 20:16:03.651229 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:03.651191    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4230221e-87ee-4ab2-9a6d-249731205b44" path="/var/lib/kubelet/pods/4230221e-87ee-4ab2-9a6d-249731205b44/volumes"
Jul 28 20:16:03.651644 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:03.651582    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c126fcc5-7f33-4239-b4a8-92bf1b3445aa" path="/var/lib/kubelet/pods/c126fcc5-7f33-4239-b4a8-92bf1b3445aa/volumes"
Jul 28 20:16:04.877901 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:04.877865    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:16:05.933784 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:05.933740    2578 patch_prober.go:28] interesting pod/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 28 20:16:05.934190 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:05.933820    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:16:06.967086 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:06.967058    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:16:07.034568 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034538    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-credential-socket\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034619    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc7xt\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-kube-api-access-fc7xt\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034649    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a30e93-fb6d-4bf4-8183-406306794334-istiod-ca-cert\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-certs\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034684    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-data\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034704    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a30e93-fb6d-4bf4-8183-406306794334-istio-podinfo\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034738    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-envoy\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.034779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034760    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-istio-token\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.035144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034801    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-socket\") pod \"23a30e93-fb6d-4bf4-8183-406306794334\" (UID: \"23a30e93-fb6d-4bf4-8183-406306794334\") "
Jul 28 20:16:07.035144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.034869    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:07.035144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.035064    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:07.035144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.035125    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.035392 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.035329    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:07.035441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.035099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23a30e93-fb6d-4bf4-8183-406306794334-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:16:07.035827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.035804    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-data" (OuterVolumeSpecName: "istio-data") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:07.037359 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.037332    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/23a30e93-fb6d-4bf4-8183-406306794334-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:16:07.037359 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.037333    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-istio-token" (OuterVolumeSpecName: "istio-token") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:07.037574 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.037555    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:16:07.037639 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.037599    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-kube-api-access-fc7xt" (OuterVolumeSpecName: "kube-api-access-fc7xt") pod "23a30e93-fb6d-4bf4-8183-406306794334" (UID: "23a30e93-fb6d-4bf4-8183-406306794334"). InnerVolumeSpecName "kube-api-access-fc7xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:16:07.135994 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.135952    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc7xt\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-kube-api-access-fc7xt\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.135994 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.135988    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/23a30e93-fb6d-4bf4-8183-406306794334-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.135994 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.136001    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.136251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.136014    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.136251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.136025    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/23a30e93-fb6d-4bf4-8183-406306794334-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.136251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.136035    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.136251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.136046    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/23a30e93-fb6d-4bf4-8183-406306794334-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.136251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.136056    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/23a30e93-fb6d-4bf4-8183-406306794334-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:16:07.467674 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.467573    2578 generic.go:356] "Generic (PLEG): container finished" podID="23a30e93-fb6d-4bf4-8183-406306794334" containerID="cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e" exitCode=0
Jul 28 20:16:07.467674 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.467628    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"
Jul 28 20:16:07.467930 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.467665    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" event={"ID":"23a30e93-fb6d-4bf4-8183-406306794334","Type":"ContainerDied","Data":"cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e"}
Jul 28 20:16:07.467930 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.467741    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh" event={"ID":"23a30e93-fb6d-4bf4-8183-406306794334","Type":"ContainerDied","Data":"59aa8fe6837a7e7668fac503f7576ded384c125a3f136acb21bb39207d6de46c"}
Jul 28 20:16:07.467930 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.467770    2578 scope.go:117] "RemoveContainer" containerID="cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e"
Jul 28 20:16:07.477947 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.477925    2578 scope.go:117] "RemoveContainer" containerID="caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"
Jul 28 20:16:07.486779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.486759    2578 scope.go:117] "RemoveContainer" containerID="cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e"
Jul 28 20:16:07.487034 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:16:07.487015    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e\": container with ID starting with cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e not found: ID does not exist" containerID="cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e"
Jul 28 20:16:07.487080 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.487045    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e"} err="failed to get container status \"cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e\": rpc error: code = NotFound desc = could not find container \"cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e\": container with ID starting with cb44cc2f258f2f2f21617dbbfd325262aa966b26bfa3a52b7159fbfb68a2b15e not found: ID does not exist"
Jul 28 20:16:07.487080 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.487063    2578 scope.go:117] "RemoveContainer" containerID="caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"
Jul 28 20:16:07.487330 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:16:07.487311    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47\": container with ID starting with caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47 not found: ID does not exist" containerID="caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"
Jul 28 20:16:07.487398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.487337    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47"} err="failed to get container status \"caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47\": rpc error: code = NotFound desc = could not find container \"caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47\": container with ID starting with caea5e0814db7ffb43b2849c23acce1d323a7502a31485931e19359f3d1aed47 not found: ID does not exist"
Jul 28 20:16:07.491239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.491215    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"]
Jul 28 20:16:07.495402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.495376    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-10cc1978-openshift-default-888dc9d44-dqmsh"]
Jul 28 20:16:07.656516 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:07.656479    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23a30e93-fb6d-4bf4-8183-406306794334" path="/var/lib/kubelet/pods/23a30e93-fb6d-4bf4-8183-406306794334/volumes"
Jul 28 20:16:37.665243 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:37.665211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:16:37.671781 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:16:37.671756    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:17:25.773823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:25.773782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:17:37.967673 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:37.967620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:17:48.261610 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:48.261573    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:17:49.984678 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.984642    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"]
Jul 28 20:17:49.985154 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985133    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985157    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985184    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985193    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985208    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985216    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985233    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy"
Jul 28 20:17:49.985239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985241    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985253    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c126fcc5-7f33-4239-b4a8-92bf1b3445aa" containerName="payload-pre-processing"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985281    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c126fcc5-7f33-4239-b4a8-92bf1b3445aa" containerName="payload-pre-processing"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985296    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4230221e-87ee-4ab2-9a6d-249731205b44" containerName="maas-api"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985304    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="4230221e-87ee-4ab2-9a6d-249731205b44" containerName="maas-api"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985391    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="4230221e-87ee-4ab2-9a6d-249731205b44" containerName="maas-api"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985407    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985416    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985427    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b380c359-3d9a-4ef3-a2c8-6eae45024292" containerName="istio-proxy"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985435    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23a30e93-fb6d-4bf4-8183-406306794334" containerName="istio-proxy"
Jul 28 20:17:49.985619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.985444    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c126fcc5-7f33-4239-b4a8-92bf1b3445aa" containerName="payload-pre-processing"
Jul 28 20:17:49.988478 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.988458    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:49.991046 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:49.991026    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6e5ce9cb-openshift-default-dockercfg-jrmv6\""
Jul 28 20:17:50.009808 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.009782    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"]
Jul 28 20:17:50.088436 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088398    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-certs\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088436 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088437    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-envoy\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088661 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088462    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-token\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088661 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088517    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-credential-socket\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088661 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088558    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-socket\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088661 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088575    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84gz4\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-kube-api-access-84gz4\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088661 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088638    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-podinfo\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088851 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-data\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.088851 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.088699    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea74217-292d-4e6b-88f7-5e95de53f37a-istiod-ca-cert\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189034    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-token\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189080    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-credential-socket\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189141    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-socket\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189174    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84gz4\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-kube-api-access-84gz4\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189214    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-podinfo\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189239    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-data\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189588 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea74217-292d-4e6b-88f7-5e95de53f37a-istiod-ca-cert\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189588 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-certs\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189588 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189370    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-envoy\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189766 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189598    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-credential-socket\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189766 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189656    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-data\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189766 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189745    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-socket\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.189925 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.189868    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-certs\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.190177 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.190063    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea74217-292d-4e6b-88f7-5e95de53f37a-istiod-ca-cert\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.191680 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.191660    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-envoy\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.191900 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.191884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-podinfo\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.197735 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.197705    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84gz4\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-kube-api-access-84gz4\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.197947 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.197932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-token\") pod \"e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") " pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.298946 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.298921    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:50.432762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.432735    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"]
Jul 28 20:17:50.434261 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:17:50.434236    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ea74217_292d_4e6b_88f7_5e95de53f37a.slice/crio-f38f346ad65bad5da86ce2669cf5ebd2971fdbc94e395ea59410a809b94f31d7 WatchSource:0}: Error finding container f38f346ad65bad5da86ce2669cf5ebd2971fdbc94e395ea59410a809b94f31d7: Status 404 returned error can't find the container with id f38f346ad65bad5da86ce2669cf5ebd2971fdbc94e395ea59410a809b94f31d7
Jul 28 20:17:50.436380 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.436346    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:17:50.436455 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.436413    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:17:50.436455 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.436447    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:17:50.985086 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.985052    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" event={"ID":"7ea74217-292d-4e6b-88f7-5e95de53f37a","Type":"ContainerStarted","Data":"a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"}
Jul 28 20:17:50.985501 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:50.985104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" event={"ID":"7ea74217-292d-4e6b-88f7-5e95de53f37a","Type":"ContainerStarted","Data":"f38f346ad65bad5da86ce2669cf5ebd2971fdbc94e395ea59410a809b94f31d7"}
Jul 28 20:17:51.015295 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:51.015219    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" podStartSLOduration=2.015200551 podStartE2EDuration="2.015200551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:51.014506128 +0000 UTC m=+1873.914722970" watchObservedRunningTime="2026-07-28 20:17:51.015200551 +0000 UTC m=+1873.915417393"
Jul 28 20:17:51.299960 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:51.299930    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:51.304706 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:51.304678    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:51.990629 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:51.990591    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:51.991723 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:51.991697    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:17:53.068756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.068717    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"]
Jul 28 20:17:53.072627 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.072606    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.075717 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.075689    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-6e5ce9cb-serving-cert\""
Jul 28 20:17:53.075858 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.075721    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:17:53.075858 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.075697    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:17:53.075858 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.075705    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:17:53.083370 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.083256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"]
Jul 28 20:17:53.111077 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.111046    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd25915-f357-49ce-9085-2da3a2b7196c-openshift-service-ca\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.111209 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.111080    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvmqb\" (UniqueName: \"kubernetes.io/projected/5bd25915-f357-49ce-9085-2da3a2b7196c-kube-api-access-tvmqb\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.111209 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.111105    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.212502 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.212463    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd25915-f357-49ce-9085-2da3a2b7196c-openshift-service-ca\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.212502 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.212504    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvmqb\" (UniqueName: \"kubernetes.io/projected/5bd25915-f357-49ce-9085-2da3a2b7196c-kube-api-access-tvmqb\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.212737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.212539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.212737 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:17:53.212673    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-serving-cert: secret "maas-api-e2e-mt-6e5ce9cb-serving-cert" not found
Jul 28 20:17:53.212819 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:17:53.212754    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls podName:5bd25915-f357-49ce-9085-2da3a2b7196c nodeName:}" failed. No retries permitted until 2026-07-28 20:17:53.712732828 +0000 UTC m=+1876.612949649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls") pod "maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" (UID: "5bd25915-f357-49ce-9085-2da3a2b7196c") : secret "maas-api-e2e-mt-6e5ce9cb-serving-cert" not found
Jul 28 20:17:53.213208 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.213188    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd25915-f357-49ce-9085-2da3a2b7196c-openshift-service-ca\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.234668 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.234631    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvmqb\" (UniqueName: \"kubernetes.io/projected/5bd25915-f357-49ce-9085-2da3a2b7196c-kube-api-access-tvmqb\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.716198 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.716164    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.718682 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.718659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls\") pod \"maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:53.984471 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:53.984398    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:54.050864 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.050833    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"]
Jul 28 20:17:54.054728 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.054706    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.057457 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.057435    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6e5ce9cb-dockercfg-nglb4\""
Jul 28 20:17:54.057457 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.057447    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6e5ce9cb\""
Jul 28 20:17:54.063904 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.063843    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"]
Jul 28 20:17:54.076062 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.076038    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"]
Jul 28 20:17:54.079437 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.079417    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.090909 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.090875    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"]
Jul 28 20:17:54.119708 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.119677    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316e0c6e-0e86-454a-a159-36d7f4cdabf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.119836 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.119716    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b42b85-a956-4b9c-8a9d-6a143231a895-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") " pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.119836 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.119735    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp5jm\" (UniqueName: \"kubernetes.io/projected/73b42b85-a956-4b9c-8a9d-6a143231a895-kube-api-access-vp5jm\") pod \"payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") " pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.119836 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.119803    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lprm4\" (UniqueName: \"kubernetes.io/projected/316e0c6e-0e86-454a-a159-36d7f4cdabf3-kube-api-access-lprm4\") pod \"payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.130223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.130200    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"]
Jul 28 20:17:54.132117 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:17:54.132089    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd25915_f357_49ce_9085_2da3a2b7196c.slice/crio-bcf75e639e12a0022a401f5487863d389be592f7d1c9bc2c5c0fa5bbc0ea4e1e WatchSource:0}: Error finding container bcf75e639e12a0022a401f5487863d389be592f7d1c9bc2c5c0fa5bbc0ea4e1e: Status 404 returned error can't find the container with id bcf75e639e12a0022a401f5487863d389be592f7d1c9bc2c5c0fa5bbc0ea4e1e
Jul 28 20:17:54.221048 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.221013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316e0c6e-0e86-454a-a159-36d7f4cdabf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.221254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.221073    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b42b85-a956-4b9c-8a9d-6a143231a895-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") " pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.221254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.221111    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp5jm\" (UniqueName: \"kubernetes.io/projected/73b42b85-a956-4b9c-8a9d-6a143231a895-kube-api-access-vp5jm\") pod \"payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") " pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.221254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.221152    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lprm4\" (UniqueName: \"kubernetes.io/projected/316e0c6e-0e86-454a-a159-36d7f4cdabf3-kube-api-access-lprm4\") pod \"payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.221832 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.221813    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316e0c6e-0e86-454a-a159-36d7f4cdabf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.221885 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.221840    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b42b85-a956-4b9c-8a9d-6a143231a895-plugins-config-volume\") pod \"payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") " pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.229713 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.229688    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp5jm\" (UniqueName: \"kubernetes.io/projected/73b42b85-a956-4b9c-8a9d-6a143231a895-kube-api-access-vp5jm\") pod \"payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") " pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.230059 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.230038    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lprm4\" (UniqueName: \"kubernetes.io/projected/316e0c6e-0e86-454a-a159-36d7f4cdabf3-kube-api-access-lprm4\") pod \"payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.367492 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.367453    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:54.393772 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.393739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:54.519259 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.519152    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"]
Jul 28 20:17:54.521665 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:17:54.521639    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316e0c6e_0e86_454a_a159_36d7f4cdabf3.slice/crio-4b6fbf9c4bf4dd6b774a38b7f87103fb0190033704be1339b3b0096278b3c8ca WatchSource:0}: Error finding container 4b6fbf9c4bf4dd6b774a38b7f87103fb0190033704be1339b3b0096278b3c8ca: Status 404 returned error can't find the container with id 4b6fbf9c4bf4dd6b774a38b7f87103fb0190033704be1339b3b0096278b3c8ca
Jul 28 20:17:54.544852 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:54.544816    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"]
Jul 28 20:17:54.557402 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:17:54.557378    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73b42b85_a956_4b9c_8a9d_6a143231a895.slice/crio-a2b2c5a3aec74aaa501976b64f6b4c997566c966ee0867596631f6b999a077e6 WatchSource:0}: Error finding container a2b2c5a3aec74aaa501976b64f6b4c997566c966ee0867596631f6b999a077e6: Status 404 returned error can't find the container with id a2b2c5a3aec74aaa501976b64f6b4c997566c966ee0867596631f6b999a077e6
Jul 28 20:17:55.006964 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.006926    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd" event={"ID":"316e0c6e-0e86-454a-a159-36d7f4cdabf3","Type":"ContainerStarted","Data":"d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6"}
Jul 28 20:17:55.007162 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.007103    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:17:55.007162 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.007129    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd" event={"ID":"316e0c6e-0e86-454a-a159-36d7f4cdabf3","Type":"ContainerStarted","Data":"4b6fbf9c4bf4dd6b774a38b7f87103fb0190033704be1339b3b0096278b3c8ca"}
Jul 28 20:17:55.008429 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.008396    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw" event={"ID":"73b42b85-a956-4b9c-8a9d-6a143231a895","Type":"ContainerStarted","Data":"807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b"}
Jul 28 20:17:55.008550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.008446    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw" event={"ID":"73b42b85-a956-4b9c-8a9d-6a143231a895","Type":"ContainerStarted","Data":"a2b2c5a3aec74aaa501976b64f6b4c997566c966ee0867596631f6b999a077e6"}
Jul 28 20:17:55.008550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.008475    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:17:55.009882 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.009830    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" event={"ID":"5bd25915-f357-49ce-9085-2da3a2b7196c","Type":"ContainerStarted","Data":"237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea"}
Jul 28 20:17:55.009882 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.009876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" event={"ID":"5bd25915-f357-49ce-9085-2da3a2b7196c","Type":"ContainerStarted","Data":"bcf75e639e12a0022a401f5487863d389be592f7d1c9bc2c5c0fa5bbc0ea4e1e"}
Jul 28 20:17:55.010027 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.009981    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:17:55.024272 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.024229    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd" podStartSLOduration=1.024214234 podStartE2EDuration="1.024214234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:55.02177369 +0000 UTC m=+1877.921990532" watchObservedRunningTime="2026-07-28 20:17:55.024214234 +0000 UTC m=+1877.924431086"
Jul 28 20:17:55.040756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.040708    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" podStartSLOduration=1.538915331 podStartE2EDuration="2.040688303s" totalImagesPullingTime="501.772972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:55.038654635 +0000 UTC m=+1877.938871477" watchObservedRunningTime="2026-07-28 20:17:55.040688303 +0000 UTC m=+1877.940905144"
Jul 28 20:17:55.054486 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:55.054447    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw" podStartSLOduration=1.054435407 podStartE2EDuration="1.054435407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:17:55.053088636 +0000 UTC m=+1877.953305488" watchObservedRunningTime="2026-07-28 20:17:55.054435407 +0000 UTC m=+1877.954652249"
Jul 28 20:17:58.368853 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:17:58.368816    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:18:01.028948 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:01.028914    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:18:06.019491 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.019444    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:18:06.019984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.019576    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:18:06.079730 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.079698    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerID="a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd" exitCode=0
Jul 28 20:18:06.079899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.079778    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" event={"ID":"7ea74217-292d-4e6b-88f7-5e95de53f37a","Type":"ContainerDied","Data":"a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"}
Jul 28 20:18:06.080193 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.080179    2578 scope.go:117] "RemoveContainer" containerID="a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"
Jul 28 20:18:06.081217 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.081193    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:18:06.081299 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.081257    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:18:06.081342 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.081321    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:18:06.991081 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:06.991032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:07.087931 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:07.087894    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" event={"ID":"7ea74217-292d-4e6b-88f7-5e95de53f37a","Type":"ContainerStarted","Data":"4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661"}
Jul 28 20:18:07.299441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:07.299387    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:07.300680 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:07.300649    2578 patch_prober.go:28] interesting pod/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 20:18:07.300803 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:07.300708    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 20:18:08.300340 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:08.300297    2578 patch_prober.go:28] interesting pod/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 20:18:08.300739 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:08.300358    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 20:18:09.300287 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:09.300222    2578 patch_prober.go:28] interesting pod/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 20:18:09.300473 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:09.300323    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 20:18:10.299417 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:10.299384    2578 patch_prober.go:28] interesting pod/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 20:18:10.299598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:10.299451    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 20:18:11.303241 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:11.303204    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:11.303711 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:11.303482    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:11.304344 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:11.304324    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:35.585248 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.585212    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"]
Jul 28 20:18:35.585836 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.585456    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" podUID="5bd25915-f357-49ce-9085-2da3a2b7196c" containerName="maas-api" containerID="cri-o://237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea" gracePeriod=30
Jul 28 20:18:35.652610 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.652576    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"]
Jul 28 20:18:35.652862 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.652835    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw" podUID="73b42b85-a956-4b9c-8a9d-6a143231a895" containerName="payload-processing" containerID="cri-o://807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b" gracePeriod=30
Jul 28 20:18:35.653376 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.653352    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"]
Jul 28 20:18:35.653627 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.653585    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd" podUID="316e0c6e-0e86-454a-a159-36d7f4cdabf3" containerName="payload-pre-processing" containerID="cri-o://d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6" gracePeriod=30
Jul 28 20:18:35.918091 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.918061    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:18:35.966322 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.966297    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:18:35.969155 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:35.969136    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:18:36.084486 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084457    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp5jm\" (UniqueName: \"kubernetes.io/projected/73b42b85-a956-4b9c-8a9d-6a143231a895-kube-api-access-vp5jm\") pod \"73b42b85-a956-4b9c-8a9d-6a143231a895\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") "
Jul 28 20:18:36.084651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084512    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls\") pod \"5bd25915-f357-49ce-9085-2da3a2b7196c\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") "
Jul 28 20:18:36.084651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084535    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b42b85-a956-4b9c-8a9d-6a143231a895-plugins-config-volume\") pod \"73b42b85-a956-4b9c-8a9d-6a143231a895\" (UID: \"73b42b85-a956-4b9c-8a9d-6a143231a895\") "
Jul 28 20:18:36.084651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084582    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316e0c6e-0e86-454a-a159-36d7f4cdabf3-plugins-config-volume\") pod \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") "
Jul 28 20:18:36.084651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084604    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lprm4\" (UniqueName: \"kubernetes.io/projected/316e0c6e-0e86-454a-a159-36d7f4cdabf3-kube-api-access-lprm4\") pod \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\" (UID: \"316e0c6e-0e86-454a-a159-36d7f4cdabf3\") "
Jul 28 20:18:36.084651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084621    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd25915-f357-49ce-9085-2da3a2b7196c-openshift-service-ca\") pod \"5bd25915-f357-49ce-9085-2da3a2b7196c\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") "
Jul 28 20:18:36.084651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084644    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvmqb\" (UniqueName: \"kubernetes.io/projected/5bd25915-f357-49ce-9085-2da3a2b7196c-kube-api-access-tvmqb\") pod \"5bd25915-f357-49ce-9085-2da3a2b7196c\" (UID: \"5bd25915-f357-49ce-9085-2da3a2b7196c\") "
Jul 28 20:18:36.084984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084961    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b42b85-a956-4b9c-8a9d-6a143231a895-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73b42b85-a956-4b9c-8a9d-6a143231a895" (UID: "73b42b85-a956-4b9c-8a9d-6a143231a895"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:36.085038 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.084969    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd25915-f357-49ce-9085-2da3a2b7196c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5bd25915-f357-49ce-9085-2da3a2b7196c" (UID: "5bd25915-f357-49ce-9085-2da3a2b7196c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:36.085196 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.085146    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316e0c6e-0e86-454a-a159-36d7f4cdabf3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "316e0c6e-0e86-454a-a159-36d7f4cdabf3" (UID: "316e0c6e-0e86-454a-a159-36d7f4cdabf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:36.086850 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.086816    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b42b85-a956-4b9c-8a9d-6a143231a895-kube-api-access-vp5jm" (OuterVolumeSpecName: "kube-api-access-vp5jm") pod "73b42b85-a956-4b9c-8a9d-6a143231a895" (UID: "73b42b85-a956-4b9c-8a9d-6a143231a895"). InnerVolumeSpecName "kube-api-access-vp5jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:36.086850 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.086841    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/316e0c6e-0e86-454a-a159-36d7f4cdabf3-kube-api-access-lprm4" (OuterVolumeSpecName: "kube-api-access-lprm4") pod "316e0c6e-0e86-454a-a159-36d7f4cdabf3" (UID: "316e0c6e-0e86-454a-a159-36d7f4cdabf3"). InnerVolumeSpecName "kube-api-access-lprm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:36.086987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.086888    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5bd25915-f357-49ce-9085-2da3a2b7196c" (UID: "5bd25915-f357-49ce-9085-2da3a2b7196c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:18:36.086987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.086918    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd25915-f357-49ce-9085-2da3a2b7196c-kube-api-access-tvmqb" (OuterVolumeSpecName: "kube-api-access-tvmqb") pod "5bd25915-f357-49ce-9085-2da3a2b7196c" (UID: "5bd25915-f357-49ce-9085-2da3a2b7196c"). InnerVolumeSpecName "kube-api-access-tvmqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185461    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5bd25915-f357-49ce-9085-2da3a2b7196c-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185488    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73b42b85-a956-4b9c-8a9d-6a143231a895-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185499    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/316e0c6e-0e86-454a-a159-36d7f4cdabf3-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185507    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lprm4\" (UniqueName: \"kubernetes.io/projected/316e0c6e-0e86-454a-a159-36d7f4cdabf3-kube-api-access-lprm4\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185516    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bd25915-f357-49ce-9085-2da3a2b7196c-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185524    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvmqb\" (UniqueName: \"kubernetes.io/projected/5bd25915-f357-49ce-9085-2da3a2b7196c-kube-api-access-tvmqb\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.185531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.185532    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp5jm\" (UniqueName: \"kubernetes.io/projected/73b42b85-a956-4b9c-8a9d-6a143231a895-kube-api-access-vp5jm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:36.245161 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.245115    2578 generic.go:356] "Generic (PLEG): container finished" podID="73b42b85-a956-4b9c-8a9d-6a143231a895" containerID="807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b" exitCode=0
Jul 28 20:18:36.245350 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.245163    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"
Jul 28 20:18:36.245350 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.245201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw" event={"ID":"73b42b85-a956-4b9c-8a9d-6a143231a895","Type":"ContainerDied","Data":"807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b"}
Jul 28 20:18:36.245350 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.245285    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw" event={"ID":"73b42b85-a956-4b9c-8a9d-6a143231a895","Type":"ContainerDied","Data":"a2b2c5a3aec74aaa501976b64f6b4c997566c966ee0867596631f6b999a077e6"}
Jul 28 20:18:36.245350 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.245303    2578 scope.go:117] "RemoveContainer" containerID="807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b"
Jul 28 20:18:36.246464 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.246440    2578 generic.go:356] "Generic (PLEG): container finished" podID="5bd25915-f357-49ce-9085-2da3a2b7196c" containerID="237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea" exitCode=0
Jul 28 20:18:36.246612 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.246485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"
Jul 28 20:18:36.246612 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.246525    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" event={"ID":"5bd25915-f357-49ce-9085-2da3a2b7196c","Type":"ContainerDied","Data":"237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea"}
Jul 28 20:18:36.246612 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.246572    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92" event={"ID":"5bd25915-f357-49ce-9085-2da3a2b7196c","Type":"ContainerDied","Data":"bcf75e639e12a0022a401f5487863d389be592f7d1c9bc2c5c0fa5bbc0ea4e1e"}
Jul 28 20:18:36.248155 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.248130    2578 generic.go:356] "Generic (PLEG): container finished" podID="316e0c6e-0e86-454a-a159-36d7f4cdabf3" containerID="d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6" exitCode=0
Jul 28 20:18:36.248258 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.248171    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"
Jul 28 20:18:36.248258 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.248203    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd" event={"ID":"316e0c6e-0e86-454a-a159-36d7f4cdabf3","Type":"ContainerDied","Data":"d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6"}
Jul 28 20:18:36.248258 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.248240    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd" event={"ID":"316e0c6e-0e86-454a-a159-36d7f4cdabf3","Type":"ContainerDied","Data":"4b6fbf9c4bf4dd6b774a38b7f87103fb0190033704be1339b3b0096278b3c8ca"}
Jul 28 20:18:36.255761 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.255744    2578 scope.go:117] "RemoveContainer" containerID="807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b"
Jul 28 20:18:36.256020 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:18:36.255997    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b\": container with ID starting with 807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b not found: ID does not exist" containerID="807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b"
Jul 28 20:18:36.256101 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.256031    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b"} err="failed to get container status \"807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b\": rpc error: code = NotFound desc = could not find container \"807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b\": container with ID starting with 807e8a76f4f7145f3782f4ca38ea0413051d91a741155003871f73040574080b not found: ID does not exist"
Jul 28 20:18:36.256101 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.256054    2578 scope.go:117] "RemoveContainer" containerID="237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea"
Jul 28 20:18:36.264618 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.264601    2578 scope.go:117] "RemoveContainer" containerID="237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea"
Jul 28 20:18:36.264850 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:18:36.264834    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea\": container with ID starting with 237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea not found: ID does not exist" containerID="237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea"
Jul 28 20:18:36.264889 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.264858    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea"} err="failed to get container status \"237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea\": rpc error: code = NotFound desc = could not find container \"237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea\": container with ID starting with 237ca0cfd64d403a9fcc5218ad44744d67f02773a361dd421a42b8c31c65cbea not found: ID does not exist"
Jul 28 20:18:36.264889 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.264880    2578 scope.go:117] "RemoveContainer" containerID="d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6"
Jul 28 20:18:36.272256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.272223    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"]
Jul 28 20:18:36.273935 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.273920    2578 scope.go:117] "RemoveContainer" containerID="d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6"
Jul 28 20:18:36.274212 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:18:36.274194    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6\": container with ID starting with d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6 not found: ID does not exist" containerID="d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6"
Jul 28 20:18:36.274308 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.274218    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6"} err="failed to get container status \"d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6\": rpc error: code = NotFound desc = could not find container \"d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6\": container with ID starting with d780425857bd206ef79f4bb5ab7b21cfc4634b6b0302264aba06b3dd020d14b6 not found: ID does not exist"
Jul 28 20:18:36.275238 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.275199    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6e5ce9cb-649c7fccc4-b2hsw"]
Jul 28 20:18:36.285164 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.285138    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"]
Jul 28 20:18:36.288403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.288385    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6e5ce9cb-56478cb56d-2g6dd"]
Jul 28 20:18:36.299174 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.299147    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"]
Jul 28 20:18:36.300993 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:36.300964    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-6e5ce9cb-77cd897f9-85t92"]
Jul 28 20:18:37.650439 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:37.650406    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316e0c6e-0e86-454a-a159-36d7f4cdabf3" path="/var/lib/kubelet/pods/316e0c6e-0e86-454a-a159-36d7f4cdabf3/volumes"
Jul 28 20:18:37.651003 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:37.650982    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bd25915-f357-49ce-9085-2da3a2b7196c" path="/var/lib/kubelet/pods/5bd25915-f357-49ce-9085-2da3a2b7196c/volumes"
Jul 28 20:18:37.651374 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:37.651356    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73b42b85-a956-4b9c-8a9d-6a143231a895" path="/var/lib/kubelet/pods/73b42b85-a956-4b9c-8a9d-6a143231a895/volumes"
Jul 28 20:18:38.561262 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:38.561223    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:18:42.220479 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:42.220443    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"]
Jul 28 20:18:42.220980 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:42.220663    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy" containerID="cri-o://4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661" gracePeriod=30
Jul 28 20:18:47.468208 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.468177    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:47.583291 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583234    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-envoy\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583291 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-certs\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583325    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84gz4\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-kube-api-access-84gz4\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583348    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea74217-292d-4e6b-88f7-5e95de53f37a-istiod-ca-cert\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583390    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-token\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583441    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-socket\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583488    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-credential-socket\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583515    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-podinfo\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583544    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-data\") pod \"7ea74217-292d-4e6b-88f7-5e95de53f37a\" (UID: \"7ea74217-292d-4e6b-88f7-5e95de53f37a\") "
Jul 28 20:18:47.583927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583611    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:18:47.583927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583753    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:18:47.583927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583773    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:18:47.583927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583887    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.583927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583907    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.583927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583921    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.584434 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.583990    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea74217-292d-4e6b-88f7-5e95de53f37a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:18:47.584624 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.584598    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:18:47.585714 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.585691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:18:47.585871 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.585839    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:18:47.586193 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.586167    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:47.586293 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.586205    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-kube-api-access-84gz4" (OuterVolumeSpecName: "kube-api-access-84gz4") pod "7ea74217-292d-4e6b-88f7-5e95de53f37a" (UID: "7ea74217-292d-4e6b-88f7-5e95de53f37a"). InnerVolumeSpecName "kube-api-access-84gz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:18:47.684988 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.684956    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.684988 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.684984    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.684988 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.684992    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.685195 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.685002    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea74217-292d-4e6b-88f7-5e95de53f37a-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.685195 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.685010    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84gz4\" (UniqueName: \"kubernetes.io/projected/7ea74217-292d-4e6b-88f7-5e95de53f37a-kube-api-access-84gz4\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:47.685195 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:47.685018    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea74217-292d-4e6b-88f7-5e95de53f37a-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:18:48.312580 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.312545    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerID="4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661" exitCode=0
Jul 28 20:18:48.312756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.312595    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"
Jul 28 20:18:48.312756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.312630    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" event={"ID":"7ea74217-292d-4e6b-88f7-5e95de53f37a","Type":"ContainerDied","Data":"4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661"}
Jul 28 20:18:48.312756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.312688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2" event={"ID":"7ea74217-292d-4e6b-88f7-5e95de53f37a","Type":"ContainerDied","Data":"f38f346ad65bad5da86ce2669cf5ebd2971fdbc94e395ea59410a809b94f31d7"}
Jul 28 20:18:48.312756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.312711    2578 scope.go:117] "RemoveContainer" containerID="4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661"
Jul 28 20:18:48.321822 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.321809    2578 scope.go:117] "RemoveContainer" containerID="a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"
Jul 28 20:18:48.330404 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.330385    2578 scope.go:117] "RemoveContainer" containerID="4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661"
Jul 28 20:18:48.330703 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:18:48.330686    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661\": container with ID starting with 4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661 not found: ID does not exist" containerID="4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661"
Jul 28 20:18:48.330775 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.330711    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661"} err="failed to get container status \"4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661\": rpc error: code = NotFound desc = could not find container \"4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661\": container with ID starting with 4a66dd7e172427ef2900a19376cbea09c0fe86c0395dbefe23b87a9ffa269661 not found: ID does not exist"
Jul 28 20:18:48.330775 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.330736    2578 scope.go:117] "RemoveContainer" containerID="a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"
Jul 28 20:18:48.330983 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:18:48.330964    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd\": container with ID starting with a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd not found: ID does not exist" containerID="a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"
Jul 28 20:18:48.331025 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.330992    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd"} err="failed to get container status \"a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd\": rpc error: code = NotFound desc = could not find container \"a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd\": container with ID starting with a1f4ae4b37c5181b07433d74df2ade2e1501173ffc85ce3c6097d10b1ed547dd not found: ID does not exist"
Jul 28 20:18:48.331531 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.331510    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"]
Jul 28 20:18:48.338790 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:48.338765    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6e5ce9cb-openshift-default-6664476dbd-5b9z2"]
Jul 28 20:18:49.650627 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:18:49.650578    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" path="/var/lib/kubelet/pods/7ea74217-292d-4e6b-88f7-5e95de53f37a/volumes"
Jul 28 20:19:02.030327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030293    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"]
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030624    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316e0c6e-0e86-454a-a159-36d7f4cdabf3" containerName="payload-pre-processing"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030636    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="316e0c6e-0e86-454a-a159-36d7f4cdabf3" containerName="payload-pre-processing"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030649    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73b42b85-a956-4b9c-8a9d-6a143231a895" containerName="payload-processing"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030657    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="73b42b85-a956-4b9c-8a9d-6a143231a895" containerName="payload-processing"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030674    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030680    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030687    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030692    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030705    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bd25915-f357-49ce-9085-2da3a2b7196c" containerName="maas-api"
Jul 28 20:19:02.030720 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030710    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bd25915-f357-49ce-9085-2da3a2b7196c" containerName="maas-api"
Jul 28 20:19:02.031111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030770    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bd25915-f357-49ce-9085-2da3a2b7196c" containerName="maas-api"
Jul 28 20:19:02.031111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030780    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="73b42b85-a956-4b9c-8a9d-6a143231a895" containerName="payload-processing"
Jul 28 20:19:02.031111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030787    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="316e0c6e-0e86-454a-a159-36d7f4cdabf3" containerName="payload-pre-processing"
Jul 28 20:19:02.031111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030793    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy"
Jul 28 20:19:02.031111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.030804    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ea74217-292d-4e6b-88f7-5e95de53f37a" containerName="istio-proxy"
Jul 28 20:19:02.035235 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.035214    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.038287 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.038239    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3474a135-openshift-default-dockercfg-5cpk2\""
Jul 28 20:19:02.045526 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.045502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"]
Jul 28 20:19:02.082957 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.082920    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-certs\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.082957 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.082959    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108d0ab-3418-4f63-a6d1-834d7c732646-istiod-ca-cert\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083178 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.082990    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-socket\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083178 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.083049    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-token\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083178 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.083079    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-data\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083178 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.083111    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vh7p\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-kube-api-access-8vh7p\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083178 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.083138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-envoy\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083435 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.083190    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-podinfo\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.083435 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.083237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-credential-socket\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184565 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-token\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184740 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184570    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-data\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184740 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vh7p\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-kube-api-access-8vh7p\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184740 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184613    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-envoy\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184740 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184636    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-podinfo\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184740 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184663    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-credential-socket\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.184740 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184721    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-certs\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184746    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108d0ab-3418-4f63-a6d1-834d7c732646-istiod-ca-cert\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.184777    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-socket\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.185009    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-data\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185236 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.185116    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-credential-socket\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185236 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.185150    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-socket\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185383 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.185242    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-certs\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.185569 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.185547    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108d0ab-3418-4f63-a6d1-834d7c732646-istiod-ca-cert\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.187190 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.187170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-envoy\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.187433 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.187415    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-podinfo\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.194081 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.194051    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-token\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.194197 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.194152    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vh7p\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-kube-api-access-8vh7p\") pod \"e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") " pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.346197 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.346103    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:02.508896 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.508868    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"]
Jul 28 20:19:02.510662 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:19:02.510633    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd108d0ab_3418_4f63_a6d1_834d7c732646.slice/crio-2a1f825f19dc8c4579cc16cf5fb9e2cff2163aac32d22e61b5fd88b9a0ffe0b5 WatchSource:0}: Error finding container 2a1f825f19dc8c4579cc16cf5fb9e2cff2163aac32d22e61b5fd88b9a0ffe0b5: Status 404 returned error can't find the container with id 2a1f825f19dc8c4579cc16cf5fb9e2cff2163aac32d22e61b5fd88b9a0ffe0b5
Jul 28 20:19:02.512685 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.512648    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:19:02.512799 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.512715    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:19:02.512799 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:02.512758    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:19:03.396243 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:03.396201    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" event={"ID":"d108d0ab-3418-4f63-a6d1-834d7c732646","Type":"ContainerStarted","Data":"9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"}
Jul 28 20:19:03.396657 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:03.396279    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" event={"ID":"d108d0ab-3418-4f63-a6d1-834d7c732646","Type":"ContainerStarted","Data":"2a1f825f19dc8c4579cc16cf5fb9e2cff2163aac32d22e61b5fd88b9a0ffe0b5"}
Jul 28 20:19:03.419372 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:03.419326    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podStartSLOduration=1.41931144 podStartE2EDuration="1.41931144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:03.419293254 +0000 UTC m=+1946.319510094" watchObservedRunningTime="2026-07-28 20:19:03.41931144 +0000 UTC m=+1946.319528263"
Jul 28 20:19:04.347339 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:04.347301    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:04.352203 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:04.352178    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:04.401566 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:04.401537    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:04.402304 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:04.402284    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:05.438928 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.438887    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"]
Jul 28 20:19:05.442491 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.442469    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.447452 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.447428    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-3474a135-serving-cert\""
Jul 28 20:19:05.447566 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.447532    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:19:05.447623 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.447576    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:19:05.447986 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.447973    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:19:05.456011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.455989    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"]
Jul 28 20:19:05.511428 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.511390    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.511617 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.511513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11018ad6-862c-4c2a-aa77-564382c84145-openshift-service-ca\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.511617 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.511561    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgn2\" (UniqueName: \"kubernetes.io/projected/11018ad6-862c-4c2a-aa77-564382c84145-kube-api-access-5bgn2\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.612757 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.612717    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11018ad6-862c-4c2a-aa77-564382c84145-openshift-service-ca\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.612757 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.612761    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgn2\" (UniqueName: \"kubernetes.io/projected/11018ad6-862c-4c2a-aa77-564382c84145-kube-api-access-5bgn2\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.612996 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.612806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.612996 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:19:05.612942    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-serving-cert: secret "maas-api-e2e-mt-3474a135-serving-cert" not found
Jul 28 20:19:05.613083 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:19:05.613013    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls podName:11018ad6-862c-4c2a-aa77-564382c84145 nodeName:}" failed. No retries permitted until 2026-07-28 20:19:06.112993699 +0000 UTC m=+1949.013210522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls") pod "maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" (UID: "11018ad6-862c-4c2a-aa77-564382c84145") : secret "maas-api-e2e-mt-3474a135-serving-cert" not found
Jul 28 20:19:05.613468 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.613449    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11018ad6-862c-4c2a-aa77-564382c84145-openshift-service-ca\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:05.622215 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:05.622190    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgn2\" (UniqueName: \"kubernetes.io/projected/11018ad6-862c-4c2a-aa77-564382c84145-kube-api-access-5bgn2\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:06.116401 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.116359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:06.118981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.118957    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls\") pod \"maas-api-e2e-mt-3474a135-68f694f4d7-26vkb\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:06.159426 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.159389    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"]
Jul 28 20:19:06.162801 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.162785    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.167087 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.167064    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-3474a135\""
Jul 28 20:19:06.167213 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.167065    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-3474a135-dockercfg-8h6bg\""
Jul 28 20:19:06.175074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.175044    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"]
Jul 28 20:19:06.188908 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.188878    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"]
Jul 28 20:19:06.192043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.192024    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.203954 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.203932    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"]
Jul 28 20:19:06.216924 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.216894    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb66h\" (UniqueName: \"kubernetes.io/projected/2001e1c4-1916-49da-aaf6-c6f2998d1c94-kube-api-access-jb66h\") pod \"payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.217057 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.216950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1464a49-5122-42ee-914c-3792261b0b16-plugins-config-volume\") pod \"payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") " pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.217057 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.216985    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2001e1c4-1916-49da-aaf6-c6f2998d1c94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.217057 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.217002    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbpz\" (UniqueName: \"kubernetes.io/projected/e1464a49-5122-42ee-914c-3792261b0b16-kube-api-access-kxbpz\") pod \"payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") " pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.317848 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.317807    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb66h\" (UniqueName: \"kubernetes.io/projected/2001e1c4-1916-49da-aaf6-c6f2998d1c94-kube-api-access-jb66h\") pod \"payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.318070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.317877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1464a49-5122-42ee-914c-3792261b0b16-plugins-config-volume\") pod \"payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") " pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.318070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.317893    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2001e1c4-1916-49da-aaf6-c6f2998d1c94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.318070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.317911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbpz\" (UniqueName: \"kubernetes.io/projected/e1464a49-5122-42ee-914c-3792261b0b16-kube-api-access-kxbpz\") pod \"payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") " pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.318724 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.318697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2001e1c4-1916-49da-aaf6-c6f2998d1c94-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.318842 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.318697    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1464a49-5122-42ee-914c-3792261b0b16-plugins-config-volume\") pod \"payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") " pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.326710 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.326684    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb66h\" (UniqueName: \"kubernetes.io/projected/2001e1c4-1916-49da-aaf6-c6f2998d1c94-kube-api-access-jb66h\") pod \"payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.326812 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.326738    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbpz\" (UniqueName: \"kubernetes.io/projected/e1464a49-5122-42ee-914c-3792261b0b16-kube-api-access-kxbpz\") pod \"payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") " pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.354744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.354714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:06.473984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.473952    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:06.503072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.503039    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:06.506669 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.506642    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"]
Jul 28 20:19:06.508237 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:19:06.508190    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11018ad6_862c_4c2a_aa77_564382c84145.slice/crio-8685b223d93f5f0e1e8edf569dae58aaa43425ebf3a151340c2f9d3a0626784e WatchSource:0}: Error finding container 8685b223d93f5f0e1e8edf569dae58aaa43425ebf3a151340c2f9d3a0626784e: Status 404 returned error can't find the container with id 8685b223d93f5f0e1e8edf569dae58aaa43425ebf3a151340c2f9d3a0626784e
Jul 28 20:19:06.509580 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.509562    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:19:06.631621 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.631586    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"]
Jul 28 20:19:06.636398 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:19:06.636369    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2001e1c4_1916_49da_aaf6_c6f2998d1c94.slice/crio-e4bdbe2c3af1c186ba3b459041a1cc4427c780929e0915ec4a6427e1edda5083 WatchSource:0}: Error finding container e4bdbe2c3af1c186ba3b459041a1cc4427c780929e0915ec4a6427e1edda5083: Status 404 returned error can't find the container with id e4bdbe2c3af1c186ba3b459041a1cc4427c780929e0915ec4a6427e1edda5083
Jul 28 20:19:06.664065 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:06.664030    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"]
Jul 28 20:19:06.670479 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:19:06.670450    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1464a49_5122_42ee_914c_3792261b0b16.slice/crio-c9c78a344e5701590371384d0a5fb74211b1bd2ff3505665a9211d01e4672fc1 WatchSource:0}: Error finding container c9c78a344e5701590371384d0a5fb74211b1bd2ff3505665a9211d01e4672fc1: Status 404 returned error can't find the container with id c9c78a344e5701590371384d0a5fb74211b1bd2ff3505665a9211d01e4672fc1
Jul 28 20:19:07.429692 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.429652    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" event={"ID":"11018ad6-862c-4c2a-aa77-564382c84145","Type":"ContainerStarted","Data":"cf1aaa8e2706894b13e3b352197a2e149df7fd39f59419714bd55a7a31b622ae"}
Jul 28 20:19:07.429878 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.429722    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:07.429878 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.429750    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" event={"ID":"11018ad6-862c-4c2a-aa77-564382c84145","Type":"ContainerStarted","Data":"8685b223d93f5f0e1e8edf569dae58aaa43425ebf3a151340c2f9d3a0626784e"}
Jul 28 20:19:07.431344 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.431314    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng" event={"ID":"e1464a49-5122-42ee-914c-3792261b0b16","Type":"ContainerStarted","Data":"4ccd406d8fb71efbfcf13a148272ce078a3acb2daccc828c45356cc3e8c8d6d3"}
Jul 28 20:19:07.431439 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.431377    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:07.431439 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.431401    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng" event={"ID":"e1464a49-5122-42ee-914c-3792261b0b16","Type":"ContainerStarted","Data":"c9c78a344e5701590371384d0a5fb74211b1bd2ff3505665a9211d01e4672fc1"}
Jul 28 20:19:07.432915 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.432886    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" event={"ID":"2001e1c4-1916-49da-aaf6-c6f2998d1c94","Type":"ContainerStarted","Data":"9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4"}
Jul 28 20:19:07.433034 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.432934    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" event={"ID":"2001e1c4-1916-49da-aaf6-c6f2998d1c94","Type":"ContainerStarted","Data":"e4bdbe2c3af1c186ba3b459041a1cc4427c780929e0915ec4a6427e1edda5083"}
Jul 28 20:19:07.433034 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.432954    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:07.451776 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.451728    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" podStartSLOduration=2.040696147 podStartE2EDuration="2.451714169s" totalImagesPullingTime="411.018022ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:07.449990746 +0000 UTC m=+1950.350207604" watchObservedRunningTime="2026-07-28 20:19:07.451714169 +0000 UTC m=+1950.351931058"
Jul 28 20:19:07.469072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.469029    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng" podStartSLOduration=1.469014784 podStartE2EDuration="1.469014784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:07.467524821 +0000 UTC m=+1950.367741676" watchObservedRunningTime="2026-07-28 20:19:07.469014784 +0000 UTC m=+1950.369231625"
Jul 28 20:19:07.483544 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:07.483499    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" podStartSLOduration=1.483486872 podStartE2EDuration="1.483486872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:19:07.481643842 +0000 UTC m=+1950.381860683" watchObservedRunningTime="2026-07-28 20:19:07.483486872 +0000 UTC m=+1950.383703714"
Jul 28 20:19:11.971529 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:11.971494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:19:13.445289 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:13.445242    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:19:18.440874 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:18.440821    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.61:9004: connect: connection refused"
Jul 28 20:19:18.441415 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:18.441395    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:19:19.962009 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:19.961974    2578 patch_prober.go:28] interesting pod/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:37296->10.132.0.59:15021: read: connection reset by peer" start-of-body=
Jul 28 20:19:19.962582 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:19.962046    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": read tcp 10.132.0.2:37296->10.132.0.59:15021: read: connection reset by peer"
Jul 28 20:19:20.504324 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:20.504288    2578 generic.go:356] "Generic (PLEG): container finished" podID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerID="9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579" exitCode=0
Jul 28 20:19:20.504495 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:20.504342    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" event={"ID":"d108d0ab-3418-4f63-a6d1-834d7c732646","Type":"ContainerDied","Data":"9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"}
Jul 28 20:19:20.504716 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:20.504704    2578 scope.go:117] "RemoveContainer" containerID="9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"
Jul 28 20:19:20.505771 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:20.505745    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:19:20.505828 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:20.505810    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:19:20.505877 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:20.505865    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:19:21.513043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:21.513007    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" event={"ID":"d108d0ab-3418-4f63-a6d1-834d7c732646","Type":"ContainerStarted","Data":"e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78"}
Jul 28 20:19:22.346803 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:22.346759    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:22.346999 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:22.346984    2578 patch_prober.go:28] interesting pod/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 20:19:22.347044 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:22.347022    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 20:19:23.346997 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:23.346957    2578 patch_prober.go:28] interesting pod/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 20:19:23.347468 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:23.347031    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 20:19:24.346671 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:24.346630    2578 patch_prober.go:28] interesting pod/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 20:19:24.346865 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:24.346694    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 20:19:25.346617 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:25.346567    2578 patch_prober.go:28] interesting pod/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 20:19:25.347037 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:25.346652    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 20:19:26.350154 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:26.350123    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:26.350663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:26.350552    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:26.351330 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:26.351308    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:19:26.475331 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:26.475289    2578 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.61:9004: connect: connection refused"
Jul 28 20:19:28.440117 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:28.440087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:19:40.581825 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:40.581786    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:19:55.223781 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:19:55.223741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:20:00.086866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.086820    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"]
Jul 28 20:20:00.092675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.092649    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:00.096138 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.095806    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9zkwl\""
Jul 28 20:20:00.097702 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.097678    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"]
Jul 28 20:20:00.180223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.180181    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be715643-a26d-4477-8b94-e725fe63c664-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3474a135-pkhz4\" (UID: \"be715643-a26d-4477-8b94-e725fe63c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:00.280842 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.280803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be715643-a26d-4477-8b94-e725fe63c664-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3474a135-pkhz4\" (UID: \"be715643-a26d-4477-8b94-e725fe63c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:00.281449 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.281431    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be715643-a26d-4477-8b94-e725fe63c664-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-3474a135-pkhz4\" (UID: \"be715643-a26d-4477-8b94-e725fe63c664\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:00.405091 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.405056    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:00.543052 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.543022    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"]
Jul 28 20:20:00.544520 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:20:00.544494    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe715643_a26d_4477_8b94_e725fe63c664.slice/crio-02a7407723d7eb216c96844e1a75cec54355ba1a8698615201991549b6504b73 WatchSource:0}: Error finding container 02a7407723d7eb216c96844e1a75cec54355ba1a8698615201991549b6504b73: Status 404 returned error can't find the container with id 02a7407723d7eb216c96844e1a75cec54355ba1a8698615201991549b6504b73
Jul 28 20:20:00.727856 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:00.727821    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4" event={"ID":"be715643-a26d-4477-8b94-e725fe63c664","Type":"ContainerStarted","Data":"02a7407723d7eb216c96844e1a75cec54355ba1a8698615201991549b6504b73"}
Jul 28 20:20:01.734742 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:01.734704    2578 generic.go:356] "Generic (PLEG): container finished" podID="be715643-a26d-4477-8b94-e725fe63c664" containerID="7a417de0c043dbd3c2670e5d10a714cfb3894f0c29d1a4a34c65b7742aa14a57" exitCode=0
Jul 28 20:20:01.735187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:01.734793    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4" event={"ID":"be715643-a26d-4477-8b94-e725fe63c664","Type":"ContainerDied","Data":"7a417de0c043dbd3c2670e5d10a714cfb3894f0c29d1a4a34c65b7742aa14a57"}
Jul 28 20:20:02.871745 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:02.871718    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:02.901606 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:02.901569    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be715643-a26d-4477-8b94-e725fe63c664-maas-api-service-ca\") pod \"be715643-a26d-4477-8b94-e725fe63c664\" (UID: \"be715643-a26d-4477-8b94-e725fe63c664\") "
Jul 28 20:20:02.901923 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:02.901897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be715643-a26d-4477-8b94-e725fe63c664-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be715643-a26d-4477-8b94-e725fe63c664" (UID: "be715643-a26d-4477-8b94-e725fe63c664"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:20:03.002734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:03.002642    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be715643-a26d-4477-8b94-e725fe63c664-maas-api-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:03.747141 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:03.747105    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"
Jul 28 20:20:03.747318 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:03.747113    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4" event={"ID":"be715643-a26d-4477-8b94-e725fe63c664","Type":"ContainerDied","Data":"02a7407723d7eb216c96844e1a75cec54355ba1a8698615201991549b6504b73"}
Jul 28 20:20:03.747318 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:03.747236    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02a7407723d7eb216c96844e1a75cec54355ba1a8698615201991549b6504b73"
Jul 28 20:20:14.275700 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:14.275657    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:20:15.623290 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.623241    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"]
Jul 28 20:20:15.623744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.623528    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" podUID="11018ad6-862c-4c2a-aa77-564382c84145" containerName="maas-api" containerID="cri-o://cf1aaa8e2706894b13e3b352197a2e149df7fd39f59419714bd55a7a31b622ae" gracePeriod=30
Jul 28 20:20:15.686367 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.686320    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"]
Jul 28 20:20:15.686592 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.686566    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng" podUID="e1464a49-5122-42ee-914c-3792261b0b16" containerName="payload-processing" containerID="cri-o://4ccd406d8fb71efbfcf13a148272ce078a3acb2daccc828c45356cc3e8c8d6d3" gracePeriod=30
Jul 28 20:20:15.697966 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.697929    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"]
Jul 28 20:20:15.698230 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.698198    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerName="payload-pre-processing" containerID="cri-o://9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4" gracePeriod=30
Jul 28 20:20:15.812589 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.812550    2578 generic.go:356] "Generic (PLEG): container finished" podID="11018ad6-862c-4c2a-aa77-564382c84145" containerID="cf1aaa8e2706894b13e3b352197a2e149df7fd39f59419714bd55a7a31b622ae" exitCode=0
Jul 28 20:20:15.812754 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.812606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" event={"ID":"11018ad6-862c-4c2a-aa77-564382c84145","Type":"ContainerDied","Data":"cf1aaa8e2706894b13e3b352197a2e149df7fd39f59419714bd55a7a31b622ae"}
Jul 28 20:20:15.815761 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.815720    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1464a49-5122-42ee-914c-3792261b0b16" containerID="4ccd406d8fb71efbfcf13a148272ce078a3acb2daccc828c45356cc3e8c8d6d3" exitCode=0
Jul 28 20:20:15.815898 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.815795    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng" event={"ID":"e1464a49-5122-42ee-914c-3792261b0b16","Type":"ContainerDied","Data":"4ccd406d8fb71efbfcf13a148272ce078a3acb2daccc828c45356cc3e8c8d6d3"}
Jul 28 20:20:15.889798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.889484    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"]
Jul 28 20:20:15.895307 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.895249    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-3474a135-pkhz4"]
Jul 28 20:20:15.981548 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:15.981516    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:20:16.002522 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.002493    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:20:16.005611 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.005589    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:20:16.011207 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.011178    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1464a49-5122-42ee-914c-3792261b0b16-plugins-config-volume\") pod \"e1464a49-5122-42ee-914c-3792261b0b16\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") "
Jul 28 20:20:16.011395 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.011323    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kxbpz\" (UniqueName: \"kubernetes.io/projected/e1464a49-5122-42ee-914c-3792261b0b16-kube-api-access-kxbpz\") pod \"e1464a49-5122-42ee-914c-3792261b0b16\" (UID: \"e1464a49-5122-42ee-914c-3792261b0b16\") "
Jul 28 20:20:16.011589 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.011567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1464a49-5122-42ee-914c-3792261b0b16-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1464a49-5122-42ee-914c-3792261b0b16" (UID: "e1464a49-5122-42ee-914c-3792261b0b16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:20:16.013606 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.013581    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1464a49-5122-42ee-914c-3792261b0b16-kube-api-access-kxbpz" (OuterVolumeSpecName: "kube-api-access-kxbpz") pod "e1464a49-5122-42ee-914c-3792261b0b16" (UID: "e1464a49-5122-42ee-914c-3792261b0b16"). InnerVolumeSpecName "kube-api-access-kxbpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:20:16.111901 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.111859    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bgn2\" (UniqueName: \"kubernetes.io/projected/11018ad6-862c-4c2a-aa77-564382c84145-kube-api-access-5bgn2\") pod \"11018ad6-862c-4c2a-aa77-564382c84145\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") "
Jul 28 20:20:16.111901 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.111912    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2001e1c4-1916-49da-aaf6-c6f2998d1c94-plugins-config-volume\") pod \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") "
Jul 28 20:20:16.112175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.111956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11018ad6-862c-4c2a-aa77-564382c84145-openshift-service-ca\") pod \"11018ad6-862c-4c2a-aa77-564382c84145\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") "
Jul 28 20:20:16.112175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.111976    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jb66h\" (UniqueName: \"kubernetes.io/projected/2001e1c4-1916-49da-aaf6-c6f2998d1c94-kube-api-access-jb66h\") pod \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\" (UID: \"2001e1c4-1916-49da-aaf6-c6f2998d1c94\") "
Jul 28 20:20:16.112175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.112005    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls\") pod \"11018ad6-862c-4c2a-aa77-564382c84145\" (UID: \"11018ad6-862c-4c2a-aa77-564382c84145\") "
Jul 28 20:20:16.112398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.112223    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxbpz\" (UniqueName: \"kubernetes.io/projected/e1464a49-5122-42ee-914c-3792261b0b16-kube-api-access-kxbpz\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.112398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.112244    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1464a49-5122-42ee-914c-3792261b0b16-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.112398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.112365    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2001e1c4-1916-49da-aaf6-c6f2998d1c94-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2001e1c4-1916-49da-aaf6-c6f2998d1c94" (UID: "2001e1c4-1916-49da-aaf6-c6f2998d1c94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:20:16.112514 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.112372    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11018ad6-862c-4c2a-aa77-564382c84145-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "11018ad6-862c-4c2a-aa77-564382c84145" (UID: "11018ad6-862c-4c2a-aa77-564382c84145"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:20:16.114385 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.114357    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2001e1c4-1916-49da-aaf6-c6f2998d1c94-kube-api-access-jb66h" (OuterVolumeSpecName: "kube-api-access-jb66h") pod "2001e1c4-1916-49da-aaf6-c6f2998d1c94" (UID: "2001e1c4-1916-49da-aaf6-c6f2998d1c94"). InnerVolumeSpecName "kube-api-access-jb66h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:20:16.114476 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.114407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11018ad6-862c-4c2a-aa77-564382c84145-kube-api-access-5bgn2" (OuterVolumeSpecName: "kube-api-access-5bgn2") pod "11018ad6-862c-4c2a-aa77-564382c84145" (UID: "11018ad6-862c-4c2a-aa77-564382c84145"). InnerVolumeSpecName "kube-api-access-5bgn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:20:16.114476 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.114414    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "11018ad6-862c-4c2a-aa77-564382c84145" (UID: "11018ad6-862c-4c2a-aa77-564382c84145"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:20:16.213343 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.213216    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11018ad6-862c-4c2a-aa77-564382c84145-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.213343 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.213250    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb66h\" (UniqueName: \"kubernetes.io/projected/2001e1c4-1916-49da-aaf6-c6f2998d1c94-kube-api-access-jb66h\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.213343 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.213260    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11018ad6-862c-4c2a-aa77-564382c84145-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.213343 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.213301    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgn2\" (UniqueName: \"kubernetes.io/projected/11018ad6-862c-4c2a-aa77-564382c84145-kube-api-access-5bgn2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.213343 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.213309    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2001e1c4-1916-49da-aaf6-c6f2998d1c94-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:20:16.823128 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.823094    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"
Jul 28 20:20:16.823643 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.823118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb" event={"ID":"11018ad6-862c-4c2a-aa77-564382c84145","Type":"ContainerDied","Data":"8685b223d93f5f0e1e8edf569dae58aaa43425ebf3a151340c2f9d3a0626784e"}
Jul 28 20:20:16.823643 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.823192    2578 scope.go:117] "RemoveContainer" containerID="cf1aaa8e2706894b13e3b352197a2e149df7fd39f59419714bd55a7a31b622ae"
Jul 28 20:20:16.824897 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.824831    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"
Jul 28 20:20:16.825146 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.824918    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng" event={"ID":"e1464a49-5122-42ee-914c-3792261b0b16","Type":"ContainerDied","Data":"c9c78a344e5701590371384d0a5fb74211b1bd2ff3505665a9211d01e4672fc1"}
Jul 28 20:20:16.826403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.826376    2578 generic.go:356] "Generic (PLEG): container finished" podID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerID="9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4" exitCode=0
Jul 28 20:20:16.826529 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.826410    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"
Jul 28 20:20:16.826529 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.826452    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" event={"ID":"2001e1c4-1916-49da-aaf6-c6f2998d1c94","Type":"ContainerDied","Data":"9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4"}
Jul 28 20:20:16.826529 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.826496    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2" event={"ID":"2001e1c4-1916-49da-aaf6-c6f2998d1c94","Type":"ContainerDied","Data":"e4bdbe2c3af1c186ba3b459041a1cc4427c780929e0915ec4a6427e1edda5083"}
Jul 28 20:20:16.834040 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.834013    2578 scope.go:117] "RemoveContainer" containerID="4ccd406d8fb71efbfcf13a148272ce078a3acb2daccc828c45356cc3e8c8d6d3"
Jul 28 20:20:16.843002 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.842983    2578 scope.go:117] "RemoveContainer" containerID="9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4"
Jul 28 20:20:16.853260 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.853222    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"]
Jul 28 20:20:16.855778 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.855757    2578 scope.go:117] "RemoveContainer" containerID="9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4"
Jul 28 20:20:16.856120 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:20:16.856094    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4\": container with ID starting with 9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4 not found: ID does not exist" containerID="9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4"
Jul 28 20:20:16.856226 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.856125    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4"} err="failed to get container status \"9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4\": rpc error: code = NotFound desc = could not find container \"9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4\": container with ID starting with 9501d0df420f728ec788ce2b5ab961b78e8eae4f3ca69d3d6ae280e57e4bdfd4 not found: ID does not exist"
Jul 28 20:20:16.856463 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.856445    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-3474a135-68f694f4d7-26vkb"]
Jul 28 20:20:16.870101 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.870064    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"]
Jul 28 20:20:16.872070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.872045    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-3474a135-65f7f94fd-lvgng"]
Jul 28 20:20:16.883206 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.883167    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"]
Jul 28 20:20:16.892410 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:16.892323    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-3474a135-7b59f8668d-plwv2"]
Jul 28 20:20:17.651041 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:17.651007    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11018ad6-862c-4c2a-aa77-564382c84145" path="/var/lib/kubelet/pods/11018ad6-862c-4c2a-aa77-564382c84145/volumes"
Jul 28 20:20:17.651384 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:17.651370    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" path="/var/lib/kubelet/pods/2001e1c4-1916-49da-aaf6-c6f2998d1c94/volumes"
Jul 28 20:20:17.651657 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:17.651646    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be715643-a26d-4477-8b94-e725fe63c664" path="/var/lib/kubelet/pods/be715643-a26d-4477-8b94-e725fe63c664/volumes"
Jul 28 20:20:17.651952 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:17.651941    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1464a49-5122-42ee-914c-3792261b0b16" path="/var/lib/kubelet/pods/e1464a49-5122-42ee-914c-3792261b0b16/volumes"
Jul 28 20:20:23.126418 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:23.126380    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:20:40.029427 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:20:40.029393    2578 scope.go:117] "RemoveContainer" containerID="1c98abcabf702a04d9f14d7f2aa4921b7703c4a2e064bb23e06495a60751a631"
Jul 28 20:21:16.815587 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:16.815547    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"]
Jul 28 20:21:16.816128 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:16.815767    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy" containerID="cri-o://e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78" gracePeriod=30
Jul 28 20:21:20.912786 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:20.912746    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:21:22.076707 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.076632    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:21:22.172442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.172404    2578 generic.go:356] "Generic (PLEG): container finished" podID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerID="e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78" exitCode=0
Jul 28 20:21:22.172649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.172457    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"
Jul 28 20:21:22.172649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.172489    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" event={"ID":"d108d0ab-3418-4f63-a6d1-834d7c732646","Type":"ContainerDied","Data":"e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78"}
Jul 28 20:21:22.172649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.172545    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp" event={"ID":"d108d0ab-3418-4f63-a6d1-834d7c732646","Type":"ContainerDied","Data":"2a1f825f19dc8c4579cc16cf5fb9e2cff2163aac32d22e61b5fd88b9a0ffe0b5"}
Jul 28 20:21:22.172649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.172567    2578 scope.go:117] "RemoveContainer" containerID="e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78"
Jul 28 20:21:22.183401 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183367    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-socket\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183556 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183425    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-token\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183556 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183484    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108d0ab-3418-4f63-a6d1-834d7c732646-istiod-ca-cert\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183556 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183513    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-envoy\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183556 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183550    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-certs\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183575    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-data\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183597    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-credential-socket\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183628    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-podinfo\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183672    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vh7p\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-kube-api-access-8vh7p\") pod \"d108d0ab-3418-4f63-a6d1-834d7c732646\" (UID: \"d108d0ab-3418-4f63-a6d1-834d7c732646\") "
Jul 28 20:21:22.183773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183675    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:21:22.184024 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:21:22.184024 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183877    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:21:22.184024 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183926    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d108d0ab-3418-4f63-a6d1-834d7c732646-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:21:22.184024 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.183993    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.184024 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.184013    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.184228 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.184027    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.184388 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.184306    2578 scope.go:117] "RemoveContainer" containerID="9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"
Jul 28 20:21:22.184792 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.184754    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-data" (OuterVolumeSpecName: "istio-data") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:21:22.186426 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.186392    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:21:22.186426 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.186407    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-token" (OuterVolumeSpecName: "istio-token") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:21:22.186744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.186716    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:21:22.186833 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.186758    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-kube-api-access-8vh7p" (OuterVolumeSpecName: "kube-api-access-8vh7p") pod "d108d0ab-3418-4f63-a6d1-834d7c732646" (UID: "d108d0ab-3418-4f63-a6d1-834d7c732646"). InnerVolumeSpecName "kube-api-access-8vh7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:21:22.203718 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.203689    2578 scope.go:117] "RemoveContainer" containerID="e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78"
Jul 28 20:21:22.204038 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:21:22.204014    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78\": container with ID starting with e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78 not found: ID does not exist" containerID="e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78"
Jul 28 20:21:22.204103 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.204047    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78"} err="failed to get container status \"e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78\": rpc error: code = NotFound desc = could not find container \"e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78\": container with ID starting with e181a5f70c8b79f23cb73f3aed700ce715a756e069528363452965e62e66fa78 not found: ID does not exist"
Jul 28 20:21:22.204103 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.204065    2578 scope.go:117] "RemoveContainer" containerID="9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"
Jul 28 20:21:22.204400 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:21:22.204378    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579\": container with ID starting with 9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579 not found: ID does not exist" containerID="9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"
Jul 28 20:21:22.204455 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.204408    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579"} err="failed to get container status \"9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579\": rpc error: code = NotFound desc = could not find container \"9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579\": container with ID starting with 9a8d00941d09e0aeb80a250027deb0219edee23aab00ee03e561914d17f7c579 not found: ID does not exist"
Jul 28 20:21:22.284985 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.284943    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d108d0ab-3418-4f63-a6d1-834d7c732646-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.284985 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.284975    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.284985 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.284984    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.284985 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.284994    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.285285 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.285002    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vh7p\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-kube-api-access-8vh7p\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.285285 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.285011    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d108d0ab-3418-4f63-a6d1-834d7c732646-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:21:22.534215 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.534177    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"]
Jul 28 20:21:22.541580 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:22.541545    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3474a135-openshift-default-8b9f6b744-vnfqp"]
Jul 28 20:21:23.651356 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:23.651318    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" path="/var/lib/kubelet/pods/d108d0ab-3418-4f63-a6d1-834d7c732646/volumes"
Jul 28 20:21:32.115680 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:32.115637    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:21:37.701836 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:37.701807    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:21:37.706884 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:37.706852    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:21:40.062078 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:21:40.062042    2578 scope.go:117] "RemoveContainer" containerID="af2105624b243c2499c6260d882b5275c7bde2026d91b9a980cdae68bceba451"
Jul 28 20:22:25.263490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263445    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"]
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263764    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263774    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263783    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1464a49-5122-42ee-914c-3792261b0b16" containerName="payload-processing"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263788    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1464a49-5122-42ee-914c-3792261b0b16" containerName="payload-processing"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263798    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11018ad6-862c-4c2a-aa77-564382c84145" containerName="maas-api"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263804    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="11018ad6-862c-4c2a-aa77-564382c84145" containerName="maas-api"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263811    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerName="payload-pre-processing"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263817    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerName="payload-pre-processing"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263828    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be715643-a26d-4477-8b94-e725fe63c664" containerName="revoke-keys"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263834    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="be715643-a26d-4477-8b94-e725fe63c664" containerName="revoke-keys"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263885    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2001e1c4-1916-49da-aaf6-c6f2998d1c94" containerName="payload-pre-processing"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263893    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263901    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="11018ad6-862c-4c2a-aa77-564382c84145" containerName="maas-api"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263907    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1464a49-5122-42ee-914c-3792261b0b16" containerName="payload-processing"
Jul 28 20:22:25.263963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.263913    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="be715643-a26d-4477-8b94-e725fe63c664" containerName="revoke-keys"
Jul 28 20:22:25.267018 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.266997    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.271231 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.271199    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-6dd19d-openshift-default-dockercfg-9lztk\""
Jul 28 20:22:25.299750 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299707    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-envoy\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.299750 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299749    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-data\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.299983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299769    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-certs\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.299983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299857    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr2zm\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-kube-api-access-zr2zm\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.299983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299909    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-credential-socket\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.299983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299930    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-podinfo\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.299983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299957    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-socket\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.300146 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.299988    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istiod-ca-cert\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.300146 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.300030    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-token\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.316774 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.316734    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"]
Jul 28 20:22:25.401462 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-envoy\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401462 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401461    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-data\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401485    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-certs\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401518    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr2zm\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-kube-api-access-zr2zm\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401548    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-credential-socket\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401566    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-podinfo\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401597    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-socket\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401621    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istiod-ca-cert\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.401738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.401661    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-token\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.402058 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.402004    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-data\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.402058 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.402020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-credential-socket\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.402145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.402101    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-certs\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.402452 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.402425    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-socket\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.402651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.402628    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istiod-ca-cert\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.403956 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.403937    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-envoy\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.404283 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.404250    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-podinfo\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.419040 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.419005    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr2zm\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-kube-api-access-zr2zm\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.420398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.420371    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-token\") pod \"e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") " pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.577853 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.577752    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:25.737651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.737620    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"]
Jul 28 20:22:25.739254 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:22:25.739218    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bfd7d67_5bcc_403c_b3d8_83eb8871c260.slice/crio-0fba258c1dfa9e5f2ef91a8fb211eff7ea6c19eb075860edbb9bca6d92086a2b WatchSource:0}: Error finding container 0fba258c1dfa9e5f2ef91a8fb211eff7ea6c19eb075860edbb9bca6d92086a2b: Status 404 returned error can't find the container with id 0fba258c1dfa9e5f2ef91a8fb211eff7ea6c19eb075860edbb9bca6d92086a2b
Jul 28 20:22:25.741300 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.741243    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:25.741402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.741342    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:25.741402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:25.741388    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:26.507987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:26.507939    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" event={"ID":"7bfd7d67-5bcc-403c-b3d8-83eb8871c260","Type":"ContainerStarted","Data":"0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"}
Jul 28 20:22:26.508497 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:26.507999    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" event={"ID":"7bfd7d67-5bcc-403c-b3d8-83eb8871c260","Type":"ContainerStarted","Data":"0fba258c1dfa9e5f2ef91a8fb211eff7ea6c19eb075860edbb9bca6d92086a2b"}
Jul 28 20:22:26.566090 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:26.566027    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podStartSLOduration=1.566008639 podStartE2EDuration="1.566008639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:26.560992766 +0000 UTC m=+2149.461209608" watchObservedRunningTime="2026-07-28 20:22:26.566008639 +0000 UTC m=+2149.466225481"
Jul 28 20:22:26.578170 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:26.578129    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:26.583210 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:26.583182    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:27.513217 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:27.513177    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:27.514286 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:27.514250    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:29.283057 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.282965    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"]
Jul 28 20:22:29.283470 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.283330    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy"
Jul 28 20:22:29.283470 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.283343    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy"
Jul 28 20:22:29.283470 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.283395    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="d108d0ab-3418-4f63-a6d1-834d7c732646" containerName="istio-proxy"
Jul 28 20:22:29.286441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.286418    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.291990 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.291968    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-6dd19d-serving-cert\""
Jul 28 20:22:29.292141 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.292000    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:22:29.292141 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.291979    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:22:29.293007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.292977    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:22:29.332080 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.332045    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb3179-3bc1-4da2-9609-28f1eb107df7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.332312 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.332136    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.332312 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.332168    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx2k\" (UniqueName: \"kubernetes.io/projected/c5fb3179-3bc1-4da2-9609-28f1eb107df7-kube-api-access-nhx2k\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.337182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.337136    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"]
Jul 28 20:22:29.433675 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.433634    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb3179-3bc1-4da2-9609-28f1eb107df7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.433895 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.433752    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.433895 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.433785    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx2k\" (UniqueName: \"kubernetes.io/projected/c5fb3179-3bc1-4da2-9609-28f1eb107df7-kube-api-access-nhx2k\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.434030 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:22:29.433939    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-serving-cert: secret "maas-api-e2e-inf-a-6dd19d-serving-cert" not found
Jul 28 20:22:29.434103 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:22:29.434067    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls podName:c5fb3179-3bc1-4da2-9609-28f1eb107df7 nodeName:}" failed. No retries permitted until 2026-07-28 20:22:29.934044142 +0000 UTC m=+2152.834260966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls") pod "maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" (UID: "c5fb3179-3bc1-4da2-9609-28f1eb107df7") : secret "maas-api-e2e-inf-a-6dd19d-serving-cert" not found
Jul 28 20:22:29.434605 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.434568    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb3179-3bc1-4da2-9609-28f1eb107df7-openshift-service-ca\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.454082 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.454045    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx2k\" (UniqueName: \"kubernetes.io/projected/c5fb3179-3bc1-4da2-9609-28f1eb107df7-kube-api-access-nhx2k\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.938145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.938088    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:29.940899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:29.940866    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls\") pod \"maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:30.198385 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:30.198279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:30.350202 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:30.350163    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"]
Jul 28 20:22:30.352333 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:22:30.352303    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fb3179_3bc1_4da2_9609_28f1eb107df7.slice/crio-aed15cc89ab2d430442e8558fe1dffc31341c426f3f50ac977d66b62445260f6 WatchSource:0}: Error finding container aed15cc89ab2d430442e8558fe1dffc31341c426f3f50ac977d66b62445260f6: Status 404 returned error can't find the container with id aed15cc89ab2d430442e8558fe1dffc31341c426f3f50ac977d66b62445260f6
Jul 28 20:22:30.534339 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:30.534295    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" event={"ID":"c5fb3179-3bc1-4da2-9609-28f1eb107df7","Type":"ContainerStarted","Data":"aed15cc89ab2d430442e8558fe1dffc31341c426f3f50ac977d66b62445260f6"}
Jul 28 20:22:30.988436 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:30.988398    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"]
Jul 28 20:22:30.991819 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:30.991797    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:30.997576 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:30.997544    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-6dd19d\""
Jul 28 20:22:31.048180 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.048138    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jsfd\" (UniqueName: \"kubernetes.io/projected/c9634238-bb2a-46cd-8437-f705642103e6-kube-api-access-4jsfd\") pod \"payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.048430 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.048209    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9634238-bb2a-46cd-8437-f705642103e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.048777 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.048745    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"]
Jul 28 20:22:31.077709 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.077664    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"]
Jul 28 20:22:31.081109 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.081085    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.110959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.110869    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-6dd19d-dockercfg-sgz74\""
Jul 28 20:22:31.138480 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.138435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"]
Jul 28 20:22:31.149568 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.149531    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jsfd\" (UniqueName: \"kubernetes.io/projected/c9634238-bb2a-46cd-8437-f705642103e6-kube-api-access-4jsfd\") pod \"payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.149790 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.149598    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9634238-bb2a-46cd-8437-f705642103e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.149790 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.149641    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cgj2\" (UniqueName: \"kubernetes.io/projected/0c4767e8-e271-4278-816c-10c6ddbaa6c0-kube-api-access-5cgj2\") pod \"payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.149790 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.149738    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4767e8-e271-4278-816c-10c6ddbaa6c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.150506 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.150483    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9634238-bb2a-46cd-8437-f705642103e6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.161597 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.161569    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jsfd\" (UniqueName: \"kubernetes.io/projected/c9634238-bb2a-46cd-8437-f705642103e6-kube-api-access-4jsfd\") pod \"payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.250632 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.250574    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4767e8-e271-4278-816c-10c6ddbaa6c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.250818 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.250671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cgj2\" (UniqueName: \"kubernetes.io/projected/0c4767e8-e271-4278-816c-10c6ddbaa6c0-kube-api-access-5cgj2\") pod \"payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.251353 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.251333    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4767e8-e271-4278-816c-10c6ddbaa6c0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.275429 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.275393    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cgj2\" (UniqueName: \"kubernetes.io/projected/0c4767e8-e271-4278-816c-10c6ddbaa6c0-kube-api-access-5cgj2\") pod \"payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.303940 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.303888    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:31.392801 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.392594    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:31.498858 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.498823    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"]
Jul 28 20:22:31.501875 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:22:31.501828    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9634238_bb2a_46cd_8437_f705642103e6.slice/crio-c381c4af8b792ddce4307434374e4b82f6b7252b4ad353428648bba38c7ae0b7 WatchSource:0}: Error finding container c381c4af8b792ddce4307434374e4b82f6b7252b4ad353428648bba38c7ae0b7: Status 404 returned error can't find the container with id c381c4af8b792ddce4307434374e4b82f6b7252b4ad353428648bba38c7ae0b7
Jul 28 20:22:31.542300 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.542238    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" event={"ID":"c5fb3179-3bc1-4da2-9609-28f1eb107df7","Type":"ContainerStarted","Data":"ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33"}
Jul 28 20:22:31.542630 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.542577    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:31.543688 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.543666    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7" event={"ID":"c9634238-bb2a-46cd-8437-f705642103e6","Type":"ContainerStarted","Data":"c381c4af8b792ddce4307434374e4b82f6b7252b4ad353428648bba38c7ae0b7"}
Jul 28 20:22:31.611382 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:22:31.611344    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c4767e8_e271_4278_816c_10c6ddbaa6c0.slice/crio-8c04246423b8ed72ae7fa82d2e27bbbf58c804242629c2f1d3f7a04ed60a564a WatchSource:0}: Error finding container 8c04246423b8ed72ae7fa82d2e27bbbf58c804242629c2f1d3f7a04ed60a564a: Status 404 returned error can't find the container with id 8c04246423b8ed72ae7fa82d2e27bbbf58c804242629c2f1d3f7a04ed60a564a
Jul 28 20:22:31.615544 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.615514    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"]
Jul 28 20:22:31.617081 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:31.617029    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" podStartSLOduration=2.126110892 podStartE2EDuration="2.617013283s" totalImagesPullingTime="490.902391ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:31.607023231 +0000 UTC m=+2154.507240073" watchObservedRunningTime="2026-07-28 20:22:31.617013283 +0000 UTC m=+2154.517230131"
Jul 28 20:22:32.552100 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.552055    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7" event={"ID":"c9634238-bb2a-46cd-8437-f705642103e6","Type":"ContainerStarted","Data":"663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a"}
Jul 28 20:22:32.552607 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.552131    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:32.553735 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.553701    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7" event={"ID":"0c4767e8-e271-4278-816c-10c6ddbaa6c0","Type":"ContainerStarted","Data":"ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034"}
Jul 28 20:22:32.553869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.553764    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:32.553869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.553789    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7" event={"ID":"0c4767e8-e271-4278-816c-10c6ddbaa6c0","Type":"ContainerStarted","Data":"8c04246423b8ed72ae7fa82d2e27bbbf58c804242629c2f1d3f7a04ed60a564a"}
Jul 28 20:22:32.593855 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.593801    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7" podStartSLOduration=2.593784503 podStartE2EDuration="2.593784503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:32.590576541 +0000 UTC m=+2155.490793382" watchObservedRunningTime="2026-07-28 20:22:32.593784503 +0000 UTC m=+2155.494001345"
Jul 28 20:22:32.631842 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:32.631778    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7" podStartSLOduration=1.631755446 podStartE2EDuration="1.631755446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:32.62747006 +0000 UTC m=+2155.527686899" watchObservedRunningTime="2026-07-28 20:22:32.631755446 +0000 UTC m=+2155.531972293"
Jul 28 20:22:36.205005 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:36.204963    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:22:37.560074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:37.560035    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:22:43.554514 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:43.554459    2578 patch_prober.go:28] interesting pod/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 28 20:22:43.554942 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:43.554536    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 28 20:22:43.561779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:43.561749    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:22:43.561942 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:43.561918    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:22:44.620678 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:44.620636    2578 generic.go:356] "Generic (PLEG): container finished" podID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerID="0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87" exitCode=0
Jul 28 20:22:44.621240 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:44.620683    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" event={"ID":"7bfd7d67-5bcc-403c-b3d8-83eb8871c260","Type":"ContainerDied","Data":"0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"}
Jul 28 20:22:44.621240 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:44.621145    2578 scope.go:117] "RemoveContainer" containerID="0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"
Jul 28 20:22:44.622245 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:44.622214    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:44.622373 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:44.622329    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:44.622438 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:44.622380    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:45.633473 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:45.633418    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" event={"ID":"7bfd7d67-5bcc-403c-b3d8-83eb8871c260","Type":"ContainerStarted","Data":"2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729"}
Jul 28 20:22:46.577995 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:46.577958    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:46.578237 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:46.578213    2578 patch_prober.go:28] interesting pod/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:22:46.578318 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:46.578262    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 20:22:47.578203 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:47.578160    2578 patch_prober.go:28] interesting pod/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:22:47.578699 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:47.578230    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 20:22:48.578702 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:48.578662    2578 patch_prober.go:28] interesting pod/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:22:48.579150 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:48.578718    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 20:22:49.578396 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:49.578349    2578 patch_prober.go:28] interesting pod/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 28 20:22:49.578605 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:49.578409    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 28 20:22:50.582179 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:50.582147    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:50.582688 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:50.582527    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:50.583346 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:50.583327    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:22:55.184346 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.184308    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"]
Jul 28 20:22:55.187483 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.187463    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.190253 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.190224    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-117350-openshift-default-dockercfg-vtdx2\""
Jul 28 20:22:55.218104 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.218070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"]
Jul 28 20:22:55.282839 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.282794    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-envoy\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.282839 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.282836    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-token\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.282856    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istiod-ca-cert\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.282987    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-certs\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.283034    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-podinfo\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.283113    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-socket\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.283152    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-data\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283447 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.283184    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5478\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-kube-api-access-q5478\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.283447 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.283220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-credential-socket\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.384671 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384627    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-podinfo\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.384867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384689    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-socket\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.384867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384806    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-data\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.384867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384844    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5478\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-kube-api-access-q5478\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385046 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-credential-socket\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385046 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384925    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-envoy\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385046 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384942    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-token\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385046 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.384967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istiod-ca-cert\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.385046    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-certs\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.385092    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-socket\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.385148    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-data\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385471 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.385432    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-certs\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385533 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.385481    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-credential-socket\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.385764 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.385741    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istiod-ca-cert\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.387194 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.387170    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-envoy\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.387441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.387422    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-podinfo\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.404103 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.404061    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5478\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-kube-api-access-q5478\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.404398 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.404378    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-token\") pod \"e2e-inf-b-117350-openshift-default-bc5c78698-2q26k\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") " pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.497625 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.497522    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:55.664137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.664113    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"]
Jul 28 20:22:55.665753 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:22:55.665721    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc59abea4_d750_4ffd_b328_c1a6d2800ec8.slice/crio-0807bcb6fa46c9cf0e0bcbfb8249b49d7bd4cc778e603f1ab4a429e188d6b0b0 WatchSource:0}: Error finding container 0807bcb6fa46c9cf0e0bcbfb8249b49d7bd4cc778e603f1ab4a429e188d6b0b0: Status 404 returned error can't find the container with id 0807bcb6fa46c9cf0e0bcbfb8249b49d7bd4cc778e603f1ab4a429e188d6b0b0
Jul 28 20:22:55.667944 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.667911    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:55.668044 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.667994    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:55.668107 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.668039    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:22:55.687956 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:55.687917    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" event={"ID":"c59abea4-d750-4ffd-b328-c1a6d2800ec8","Type":"ContainerStarted","Data":"0807bcb6fa46c9cf0e0bcbfb8249b49d7bd4cc778e603f1ab4a429e188d6b0b0"}
Jul 28 20:22:56.695158 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:56.695118    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" event={"ID":"c59abea4-d750-4ffd-b328-c1a6d2800ec8","Type":"ContainerStarted","Data":"7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"}
Jul 28 20:22:56.745871 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:56.745814    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podStartSLOduration=1.745797032 podStartE2EDuration="1.745797032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:22:56.744891469 +0000 UTC m=+2179.645108315" watchObservedRunningTime="2026-07-28 20:22:56.745797032 +0000 UTC m=+2179.646013873"
Jul 28 20:22:57.497882 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:57.497841    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:57.503214 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:57.503183    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:57.700628 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:57.700599    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:22:57.701704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:22:57.701685    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:23:01.276167 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.276125    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"]
Jul 28 20:23:01.279765 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.279748    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.283358 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.283329    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-117350\""
Jul 28 20:23:01.283517 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.283342    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-117350-dockercfg-zgs49\""
Jul 28 20:23:01.324869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.324784    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"]
Jul 28 20:23:01.333571 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.333531    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhsjf\" (UniqueName: \"kubernetes.io/projected/8f3c6d7c-001a-4655-9b60-adf86046f5db-kube-api-access-zhsjf\") pod \"payload-processing-e2e-inf-b-117350-86d66796bb-2r96v\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") " pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.333822 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.333664    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3c6d7c-001a-4655-9b60-adf86046f5db-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-117350-86d66796bb-2r96v\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") " pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.435072 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.435022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3c6d7c-001a-4655-9b60-adf86046f5db-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-117350-86d66796bb-2r96v\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") " pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.435308 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.435092    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhsjf\" (UniqueName: \"kubernetes.io/projected/8f3c6d7c-001a-4655-9b60-adf86046f5db-kube-api-access-zhsjf\") pod \"payload-processing-e2e-inf-b-117350-86d66796bb-2r96v\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") " pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.435852 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.435830    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3c6d7c-001a-4655-9b60-adf86046f5db-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-117350-86d66796bb-2r96v\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") " pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.452406 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.452370    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhsjf\" (UniqueName: \"kubernetes.io/projected/8f3c6d7c-001a-4655-9b60-adf86046f5db-kube-api-access-zhsjf\") pod \"payload-processing-e2e-inf-b-117350-86d66796bb-2r96v\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") " pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.591402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.591297    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:01.763093 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:01.762881    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"]
Jul 28 20:23:01.765525 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:23:01.765479    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f3c6d7c_001a_4655_9b60_adf86046f5db.slice/crio-8c443a5a1640714c32b66bd75f795af6363c386ff11f540b4ef5f6db67f09499 WatchSource:0}: Error finding container 8c443a5a1640714c32b66bd75f795af6363c386ff11f540b4ef5f6db67f09499: Status 404 returned error can't find the container with id 8c443a5a1640714c32b66bd75f795af6363c386ff11f540b4ef5f6db67f09499
Jul 28 20:23:02.727980 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:02.727932    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v" event={"ID":"8f3c6d7c-001a-4655-9b60-adf86046f5db","Type":"ContainerStarted","Data":"e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f"}
Jul 28 20:23:02.728420 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:02.728002    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v" event={"ID":"8f3c6d7c-001a-4655-9b60-adf86046f5db","Type":"ContainerStarted","Data":"8c443a5a1640714c32b66bd75f795af6363c386ff11f540b4ef5f6db67f09499"}
Jul 28 20:23:02.728420 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:02.728032    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:02.761108 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:02.761053    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v" podStartSLOduration=1.761034323 podStartE2EDuration="1.761034323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:23:02.75789025 +0000 UTC m=+2185.658107092" watchObservedRunningTime="2026-07-28 20:23:02.761034323 +0000 UTC m=+2185.661251165"
Jul 28 20:23:04.504912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:04.504876    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:23:12.701902 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.701796    2578 patch_prober.go:28] interesting pod/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:23:12.701902 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.701875    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 28 20:23:12.784377 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.784344    2578 generic.go:356] "Generic (PLEG): container finished" podID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerID="7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558" exitCode=0
Jul 28 20:23:12.784553 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.784385    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" event={"ID":"c59abea4-d750-4ffd-b328-c1a6d2800ec8","Type":"ContainerDied","Data":"7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"}
Jul 28 20:23:12.784839 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.784820    2578 scope.go:117] "RemoveContainer" containerID="7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"
Jul 28 20:23:12.785930 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.785897    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:23:12.786035 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.785984    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:23:12.786074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:12.786042    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:23:13.736760 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:13.736723    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:23:13.793031 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:13.792987    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" event={"ID":"c59abea4-d750-4ffd-b328-c1a6d2800ec8","Type":"ContainerStarted","Data":"d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6"}
Jul 28 20:23:14.498537 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:14.498493    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:23:14.498770 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:14.498752    2578 patch_prober.go:28] interesting pod/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:23:14.498829 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:14.498809    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 28 20:23:15.498862 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:15.498822    2578 patch_prober.go:28] interesting pod/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:23:15.499334 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:15.498900    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 28 20:23:16.498384 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:16.498346    2578 patch_prober.go:28] interesting pod/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:23:16.498583 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:16.498408    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 28 20:23:17.498483 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:17.498442    2578 patch_prober.go:28] interesting pod/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Jul 28 20:23:17.498916 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:17.498512    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Jul 28 20:23:18.502088 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:18.502054    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:23:18.502604 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:18.502490    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:23:18.503162 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:18.503140    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:23:38.183969 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:38.183933    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:23:49.599322 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:23:49.599279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:24:22.998741 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:24:22.998702    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:24:37.098291 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:24:37.098227    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:25:14.376204 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.376154    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:25:14.716677 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.716577    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"]
Jul 28 20:25:14.716860 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.716826    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" podUID="c5fb3179-3bc1-4da2-9609-28f1eb107df7" containerName="maas-api" containerID="cri-o://ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33" gracePeriod=30
Jul 28 20:25:14.779993 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.779952    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"]
Jul 28 20:25:14.780258 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.780202    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7" podUID="c9634238-bb2a-46cd-8437-f705642103e6" containerName="payload-processing" containerID="cri-o://663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a" gracePeriod=30
Jul 28 20:25:14.788567 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.788532    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"]
Jul 28 20:25:14.788814 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:14.788789    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7" podUID="0c4767e8-e271-4278-816c-10c6ddbaa6c0" containerName="payload-pre-processing" containerID="cri-o://ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034" gracePeriod=30
Jul 28 20:25:15.067428 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.067398    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:25:15.080490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.080456    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:25:15.092537 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.092509    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:25:15.096486 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096458    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls\") pod \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") "
Jul 28 20:25:15.096647 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096511    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4767e8-e271-4278-816c-10c6ddbaa6c0-plugins-config-volume\") pod \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") "
Jul 28 20:25:15.096647 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096545    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4jsfd\" (UniqueName: \"kubernetes.io/projected/c9634238-bb2a-46cd-8437-f705642103e6-kube-api-access-4jsfd\") pod \"c9634238-bb2a-46cd-8437-f705642103e6\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") "
Jul 28 20:25:15.096647 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096593    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cgj2\" (UniqueName: \"kubernetes.io/projected/0c4767e8-e271-4278-816c-10c6ddbaa6c0-kube-api-access-5cgj2\") pod \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\" (UID: \"0c4767e8-e271-4278-816c-10c6ddbaa6c0\") "
Jul 28 20:25:15.096827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096667    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9634238-bb2a-46cd-8437-f705642103e6-plugins-config-volume\") pod \"c9634238-bb2a-46cd-8437-f705642103e6\" (UID: \"c9634238-bb2a-46cd-8437-f705642103e6\") "
Jul 28 20:25:15.096827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096713    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb3179-3bc1-4da2-9609-28f1eb107df7-openshift-service-ca\") pod \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") "
Jul 28 20:25:15.096827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096742    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhx2k\" (UniqueName: \"kubernetes.io/projected/c5fb3179-3bc1-4da2-9609-28f1eb107df7-kube-api-access-nhx2k\") pod \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\" (UID: \"c5fb3179-3bc1-4da2-9609-28f1eb107df7\") "
Jul 28 20:25:15.097031 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.096893    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c4767e8-e271-4278-816c-10c6ddbaa6c0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0c4767e8-e271-4278-816c-10c6ddbaa6c0" (UID: "0c4767e8-e271-4278-816c-10c6ddbaa6c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:25:15.097098 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.097041    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9634238-bb2a-46cd-8437-f705642103e6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c9634238-bb2a-46cd-8437-f705642103e6" (UID: "c9634238-bb2a-46cd-8437-f705642103e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:25:15.097504 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.097466    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fb3179-3bc1-4da2-9609-28f1eb107df7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5fb3179-3bc1-4da2-9609-28f1eb107df7" (UID: "c5fb3179-3bc1-4da2-9609-28f1eb107df7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:25:15.098875 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.098854    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5fb3179-3bc1-4da2-9609-28f1eb107df7" (UID: "c5fb3179-3bc1-4da2-9609-28f1eb107df7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:25:15.098960 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.098897    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9634238-bb2a-46cd-8437-f705642103e6-kube-api-access-4jsfd" (OuterVolumeSpecName: "kube-api-access-4jsfd") pod "c9634238-bb2a-46cd-8437-f705642103e6" (UID: "c9634238-bb2a-46cd-8437-f705642103e6"). InnerVolumeSpecName "kube-api-access-4jsfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:15.099514 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.099491    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c4767e8-e271-4278-816c-10c6ddbaa6c0-kube-api-access-5cgj2" (OuterVolumeSpecName: "kube-api-access-5cgj2") pod "0c4767e8-e271-4278-816c-10c6ddbaa6c0" (UID: "0c4767e8-e271-4278-816c-10c6ddbaa6c0"). InnerVolumeSpecName "kube-api-access-5cgj2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:15.099631 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.099545    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fb3179-3bc1-4da2-9609-28f1eb107df7-kube-api-access-nhx2k" (OuterVolumeSpecName: "kube-api-access-nhx2k") pod "c5fb3179-3bc1-4da2-9609-28f1eb107df7" (UID: "c5fb3179-3bc1-4da2-9609-28f1eb107df7"). InnerVolumeSpecName "kube-api-access-nhx2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:15.197662 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197618    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5fb3179-3bc1-4da2-9609-28f1eb107df7-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.197662 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197652    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhx2k\" (UniqueName: \"kubernetes.io/projected/c5fb3179-3bc1-4da2-9609-28f1eb107df7-kube-api-access-nhx2k\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.197662 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197663    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5fb3179-3bc1-4da2-9609-28f1eb107df7-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.197662 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197672    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c4767e8-e271-4278-816c-10c6ddbaa6c0-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.197947 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197683    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jsfd\" (UniqueName: \"kubernetes.io/projected/c9634238-bb2a-46cd-8437-f705642103e6-kube-api-access-4jsfd\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.197947 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197692    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cgj2\" (UniqueName: \"kubernetes.io/projected/0c4767e8-e271-4278-816c-10c6ddbaa6c0-kube-api-access-5cgj2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.197947 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.197702    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9634238-bb2a-46cd-8437-f705642103e6-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:15.456987 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.456956    2578 generic.go:356] "Generic (PLEG): container finished" podID="c5fb3179-3bc1-4da2-9609-28f1eb107df7" containerID="ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33" exitCode=0
Jul 28 20:25:15.457477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.457002    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"
Jul 28 20:25:15.457477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.457048    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" event={"ID":"c5fb3179-3bc1-4da2-9609-28f1eb107df7","Type":"ContainerDied","Data":"ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33"}
Jul 28 20:25:15.457477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.457099    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2" event={"ID":"c5fb3179-3bc1-4da2-9609-28f1eb107df7","Type":"ContainerDied","Data":"aed15cc89ab2d430442e8558fe1dffc31341c426f3f50ac977d66b62445260f6"}
Jul 28 20:25:15.457477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.457129    2578 scope.go:117] "RemoveContainer" containerID="ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33"
Jul 28 20:25:15.458825 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.458797    2578 generic.go:356] "Generic (PLEG): container finished" podID="c9634238-bb2a-46cd-8437-f705642103e6" containerID="663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a" exitCode=0
Jul 28 20:25:15.459116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.458835    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"
Jul 28 20:25:15.459116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.458946    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7" event={"ID":"c9634238-bb2a-46cd-8437-f705642103e6","Type":"ContainerDied","Data":"663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a"}
Jul 28 20:25:15.459116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.458993    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7" event={"ID":"c9634238-bb2a-46cd-8437-f705642103e6","Type":"ContainerDied","Data":"c381c4af8b792ddce4307434374e4b82f6b7252b4ad353428648bba38c7ae0b7"}
Jul 28 20:25:15.460630 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.460600    2578 generic.go:356] "Generic (PLEG): container finished" podID="0c4767e8-e271-4278-816c-10c6ddbaa6c0" containerID="ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034" exitCode=0
Jul 28 20:25:15.460767 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.460654    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"
Jul 28 20:25:15.460767 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.460668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7" event={"ID":"0c4767e8-e271-4278-816c-10c6ddbaa6c0","Type":"ContainerDied","Data":"ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034"}
Jul 28 20:25:15.460767 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.460708    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7" event={"ID":"0c4767e8-e271-4278-816c-10c6ddbaa6c0","Type":"ContainerDied","Data":"8c04246423b8ed72ae7fa82d2e27bbbf58c804242629c2f1d3f7a04ed60a564a"}
Jul 28 20:25:15.470324 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.470295    2578 scope.go:117] "RemoveContainer" containerID="ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33"
Jul 28 20:25:15.470714 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:15.470685    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33\": container with ID starting with ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33 not found: ID does not exist" containerID="ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33"
Jul 28 20:25:15.470782 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.470729    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33"} err="failed to get container status \"ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33\": rpc error: code = NotFound desc = could not find container \"ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33\": container with ID starting with ccd15389fe4abe0ea4e3d55cbdace5074af64ffed0a43b8fdcb03860ad5b0b33 not found: ID does not exist"
Jul 28 20:25:15.470782 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.470756    2578 scope.go:117] "RemoveContainer" containerID="663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a"
Jul 28 20:25:15.483941 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.483918    2578 scope.go:117] "RemoveContainer" containerID="663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a"
Jul 28 20:25:15.484357 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:15.484322    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a\": container with ID starting with 663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a not found: ID does not exist" containerID="663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a"
Jul 28 20:25:15.484478 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.484368    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a"} err="failed to get container status \"663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a\": rpc error: code = NotFound desc = could not find container \"663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a\": container with ID starting with 663f1da0e160ed723db0b81b72196c81babd1f9503f45a136150db7c6e5eab1a not found: ID does not exist"
Jul 28 20:25:15.484478 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.484397    2578 scope.go:117] "RemoveContainer" containerID="ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034"
Jul 28 20:25:15.493924 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.493897    2578 scope.go:117] "RemoveContainer" containerID="ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034"
Jul 28 20:25:15.494238 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:15.494217    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034\": container with ID starting with ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034 not found: ID does not exist" containerID="ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034"
Jul 28 20:25:15.494393 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.494248    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034"} err="failed to get container status \"ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034\": rpc error: code = NotFound desc = could not find container \"ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034\": container with ID starting with ad8443758bbda09f4388fde03001e625ec04233d6ec3a33c3b81515108192034 not found: ID does not exist"
Jul 28 20:25:15.500903 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.500863    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"]
Jul 28 20:25:15.506182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.506153    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-6dd19d-5775679b6b-56wt7"]
Jul 28 20:25:15.519867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.519833    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"]
Jul 28 20:25:15.527637 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.527605    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-6dd19d-5b4cbbbf95-5sxb2"]
Jul 28 20:25:15.542115 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.542043    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"]
Jul 28 20:25:15.544926 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.544895    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-6dd19d-649db466cb-lbvx7"]
Jul 28 20:25:15.650951 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.650907    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c4767e8-e271-4278-816c-10c6ddbaa6c0" path="/var/lib/kubelet/pods/0c4767e8-e271-4278-816c-10c6ddbaa6c0/volumes"
Jul 28 20:25:15.651417 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.651395    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5fb3179-3bc1-4da2-9609-28f1eb107df7" path="/var/lib/kubelet/pods/c5fb3179-3bc1-4da2-9609-28f1eb107df7/volumes"
Jul 28 20:25:15.651773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:15.651756    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9634238-bb2a-46cd-8437-f705642103e6" path="/var/lib/kubelet/pods/c9634238-bb2a-46cd-8437-f705642103e6/volumes"
Jul 28 20:25:26.887963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:26.887921    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:25:27.584355 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:27.584311    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"]
Jul 28 20:25:27.584626 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:27.584576    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy" containerID="cri-o://2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729" gracePeriod=30
Jul 28 20:25:32.837174 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.837145    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:25:32.950912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.950822    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-data\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.950912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.950866    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-credential-socket\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.950912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.950911    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr2zm\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-kube-api-access-zr2zm\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.950932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-certs\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.950968    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-token\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951012    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istiod-ca-cert\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951060    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-socket\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951094    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-podinfo\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951135    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-envoy\") pod \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\" (UID: \"7bfd7d67-5bcc-403c-b3d8-83eb8871c260\") "
Jul 28 20:25:32.951509 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951228    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:32.951509 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951242    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:32.951509 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951463    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:32.951509 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951480    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:32.951727 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:25:32.951790 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951739    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:32.951869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.951822    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-data" (OuterVolumeSpecName: "istio-data") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:32.953909 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.953879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:25:32.954030 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.953879    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:32.954030 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.953959    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-token" (OuterVolumeSpecName: "istio-token") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:32.954030 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:32.953972    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-kube-api-access-zr2zm" (OuterVolumeSpecName: "kube-api-access-zr2zm") pod "7bfd7d67-5bcc-403c-b3d8-83eb8871c260" (UID: "7bfd7d67-5bcc-403c-b3d8-83eb8871c260"). InnerVolumeSpecName "kube-api-access-zr2zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:33.051899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051858    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.051899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051892    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.051899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051902    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.052176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051911    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.052176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051919    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.052176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051928    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr2zm\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-kube-api-access-zr2zm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.052176 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.051937    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bfd7d67-5bcc-403c-b3d8-83eb8871c260-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:33.560439 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.560400    2578 generic.go:356] "Generic (PLEG): container finished" podID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerID="2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729" exitCode=0
Jul 28 20:25:33.560655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.560460    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"
Jul 28 20:25:33.560655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.560485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" event={"ID":"7bfd7d67-5bcc-403c-b3d8-83eb8871c260","Type":"ContainerDied","Data":"2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729"}
Jul 28 20:25:33.560655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.560542    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh" event={"ID":"7bfd7d67-5bcc-403c-b3d8-83eb8871c260","Type":"ContainerDied","Data":"0fba258c1dfa9e5f2ef91a8fb211eff7ea6c19eb075860edbb9bca6d92086a2b"}
Jul 28 20:25:33.560655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.560566    2578 scope.go:117] "RemoveContainer" containerID="2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729"
Jul 28 20:25:33.570769 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.570743    2578 scope.go:117] "RemoveContainer" containerID="0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"
Jul 28 20:25:33.579653 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.579630    2578 scope.go:117] "RemoveContainer" containerID="2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729"
Jul 28 20:25:33.579970 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:33.579950    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729\": container with ID starting with 2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729 not found: ID does not exist" containerID="2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729"
Jul 28 20:25:33.580021 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.579982    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729"} err="failed to get container status \"2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729\": rpc error: code = NotFound desc = could not find container \"2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729\": container with ID starting with 2d6b16f1743358a95484b2bbcf881df63c5b0f5e2476f07f7b80532d9ba79729 not found: ID does not exist"
Jul 28 20:25:33.580021 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.580004    2578 scope.go:117] "RemoveContainer" containerID="0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"
Jul 28 20:25:33.580257 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:33.580240    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87\": container with ID starting with 0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87 not found: ID does not exist" containerID="0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"
Jul 28 20:25:33.580328 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.580262    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87"} err="failed to get container status \"0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87\": rpc error: code = NotFound desc = could not find container \"0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87\": container with ID starting with 0b0ad08da23d27b83e8e4bde99b967e8b8de4ecb30913c105b76c7da10eb4f87 not found: ID does not exist"
Jul 28 20:25:33.587655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.587622    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"]
Jul 28 20:25:33.592085 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.592053    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-6dd19d-openshift-default-5c4984bc9d-mqwlh"]
Jul 28 20:25:33.651056 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:33.651016    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" path="/var/lib/kubelet/pods/7bfd7d67-5bcc-403c-b3d8-83eb8871c260/volumes"
Jul 28 20:25:38.604894 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.604854    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"]
Jul 28 20:25:38.605332 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.605088    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v" podUID="8f3c6d7c-001a-4655-9b60-adf86046f5db" containerName="payload-processing" containerID="cri-o://e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f" gracePeriod=30
Jul 28 20:25:38.858027 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.857954    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:25:38.901735 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.901693    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3c6d7c-001a-4655-9b60-adf86046f5db-plugins-config-volume\") pod \"8f3c6d7c-001a-4655-9b60-adf86046f5db\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") "
Jul 28 20:25:38.901912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.901751    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhsjf\" (UniqueName: \"kubernetes.io/projected/8f3c6d7c-001a-4655-9b60-adf86046f5db-kube-api-access-zhsjf\") pod \"8f3c6d7c-001a-4655-9b60-adf86046f5db\" (UID: \"8f3c6d7c-001a-4655-9b60-adf86046f5db\") "
Jul 28 20:25:38.902192 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.902159    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3c6d7c-001a-4655-9b60-adf86046f5db-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f3c6d7c-001a-4655-9b60-adf86046f5db" (UID: "8f3c6d7c-001a-4655-9b60-adf86046f5db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:25:38.904076 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:38.904053    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3c6d7c-001a-4655-9b60-adf86046f5db-kube-api-access-zhsjf" (OuterVolumeSpecName: "kube-api-access-zhsjf") pod "8f3c6d7c-001a-4655-9b60-adf86046f5db" (UID: "8f3c6d7c-001a-4655-9b60-adf86046f5db"). InnerVolumeSpecName "kube-api-access-zhsjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:39.003130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.003087    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3c6d7c-001a-4655-9b60-adf86046f5db-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:39.003130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.003121    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhsjf\" (UniqueName: \"kubernetes.io/projected/8f3c6d7c-001a-4655-9b60-adf86046f5db-kube-api-access-zhsjf\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:39.595714 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.595675    2578 generic.go:356] "Generic (PLEG): container finished" podID="8f3c6d7c-001a-4655-9b60-adf86046f5db" containerID="e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f" exitCode=0
Jul 28 20:25:39.595929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.595725    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"
Jul 28 20:25:39.595929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.595716    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v" event={"ID":"8f3c6d7c-001a-4655-9b60-adf86046f5db","Type":"ContainerDied","Data":"e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f"}
Jul 28 20:25:39.595929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.595876    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v" event={"ID":"8f3c6d7c-001a-4655-9b60-adf86046f5db","Type":"ContainerDied","Data":"8c443a5a1640714c32b66bd75f795af6363c386ff11f540b4ef5f6db67f09499"}
Jul 28 20:25:39.595929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.595901    2578 scope.go:117] "RemoveContainer" containerID="e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f"
Jul 28 20:25:39.605619 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.605597    2578 scope.go:117] "RemoveContainer" containerID="e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f"
Jul 28 20:25:39.605936 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:39.605918    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f\": container with ID starting with e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f not found: ID does not exist" containerID="e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f"
Jul 28 20:25:39.605978 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.605947    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f"} err="failed to get container status \"e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f\": rpc error: code = NotFound desc = could not find container \"e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f\": container with ID starting with e2b22d3bc07c0d64c1811d51ce1c73e9b2cbbf90ab561f0167894cf516e1a04f not found: ID does not exist"
Jul 28 20:25:39.617403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.617367    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"]
Jul 28 20:25:39.625144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.625114    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-117350-86d66796bb-2r96v"]
Jul 28 20:25:39.650728 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:39.650700    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f3c6d7c-001a-4655-9b60-adf86046f5db" path="/var/lib/kubelet/pods/8f3c6d7c-001a-4655-9b60-adf86046f5db/volumes"
Jul 28 20:25:41.776632 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:41.776591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:25:45.790256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:45.790220    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"]
Jul 28 20:25:45.790784 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:45.790529    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" containerID="cri-o://d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6" gracePeriod=30
Jul 28 20:25:47.360187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360145    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"]
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9634238-bb2a-46cd-8437-f705642103e6" containerName="payload-processing"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360527    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9634238-bb2a-46cd-8437-f705642103e6" containerName="payload-processing"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360542    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fb3179-3bc1-4da2-9609-28f1eb107df7" containerName="maas-api"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360551    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fb3179-3bc1-4da2-9609-28f1eb107df7" containerName="maas-api"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360569    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360578    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360589    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f3c6d7c-001a-4655-9b60-adf86046f5db" containerName="payload-processing"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360597    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f3c6d7c-001a-4655-9b60-adf86046f5db" containerName="payload-processing"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360613    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy"
Jul 28 20:25:47.360616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360631    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c4767e8-e271-4278-816c-10c6ddbaa6c0" containerName="payload-pre-processing"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360640    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c4767e8-e271-4278-816c-10c6ddbaa6c0" containerName="payload-pre-processing"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360744    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5fb3179-3bc1-4da2-9609-28f1eb107df7" containerName="maas-api"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360760    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9634238-bb2a-46cd-8437-f705642103e6" containerName="payload-processing"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c4767e8-e271-4278-816c-10c6ddbaa6c0" containerName="payload-pre-processing"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360776    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360784    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bfd7d67-5bcc-403c-b3d8-83eb8871c260" containerName="istio-proxy"
Jul 28 20:25:47.361007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.360792    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f3c6d7c-001a-4655-9b60-adf86046f5db" containerName="payload-processing"
Jul 28 20:25:47.362796 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.362778    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.365547 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.365523    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-711431-openshift-default-dockercfg-4lfh6\""
Jul 28 20:25:47.376151 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.376122    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"]
Jul 28 20:25:47.477126 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477093    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-socket\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477301 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477131    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-data\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477301 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477170    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-envoy\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477301 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477232    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-certs\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477301 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477254    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-credential-socket\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477301 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477292    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9x7m\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-kube-api-access-f9x7m\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-token\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477344    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f5574-851d-4692-84a8-a0b9ab767b49-istiod-ca-cert\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.477490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.477363    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-podinfo\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578466 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578424    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-socket\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578466 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-data\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578498    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-envoy\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578539    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-certs\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578571    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-credential-socket\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578738    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9x7m\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-kube-api-access-f9x7m\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578800    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-token\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578831    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f5574-851d-4692-84a8-a0b9ab767b49-istiod-ca-cert\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578834    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-socket\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578857    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-podinfo\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.578919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.578886    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-credential-socket\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.579221 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.579040    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-certs\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.579221 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.579088    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-data\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.579467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.579448    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f5574-851d-4692-84a8-a0b9ab767b49-istiod-ca-cert\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.581103 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.581083    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-envoy\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.581444 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.581427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-podinfo\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.597780 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.597743    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-token\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.602799 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.602764    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9x7m\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-kube-api-access-f9x7m\") pod \"e2e-api-a-711431-openshift-default-6bc756c986-w6d6v\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") " pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.673644 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.673542    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:47.822593 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.822566    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"]
Jul 28 20:25:47.823999 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:25:47.823967    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52f5574_851d_4692_84a8_a0b9ab767b49.slice/crio-9ba22e6151d3ec8dd6bcba9b7a72e5e9d0031bdae723a32799c925fb1a6f9cf9 WatchSource:0}: Error finding container 9ba22e6151d3ec8dd6bcba9b7a72e5e9d0031bdae723a32799c925fb1a6f9cf9: Status 404 returned error can't find the container with id 9ba22e6151d3ec8dd6bcba9b7a72e5e9d0031bdae723a32799c925fb1a6f9cf9
Jul 28 20:25:47.826165 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.826131    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:25:47.826222 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.826203    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:25:47.826259 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:47.826230    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:25:48.505484 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.505440    2578 patch_prober.go:28] interesting pod/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:25:48.505851 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.505505    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:25:48.646420 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.646373    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" event={"ID":"c52f5574-851d-4692-84a8-a0b9ab767b49","Type":"ContainerStarted","Data":"6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"}
Jul 28 20:25:48.646612 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.646437    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" event={"ID":"c52f5574-851d-4692-84a8-a0b9ab767b49","Type":"ContainerStarted","Data":"9ba22e6151d3ec8dd6bcba9b7a72e5e9d0031bdae723a32799c925fb1a6f9cf9"}
Jul 28 20:25:48.673854 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.673814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:48.675334 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.675286    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podStartSLOduration=1.675251024 podStartE2EDuration="1.675251024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:25:48.673001471 +0000 UTC m=+2351.573218338" watchObservedRunningTime="2026-07-28 20:25:48.675251024 +0000 UTC m=+2351.575467866"
Jul 28 20:25:48.679159 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:48.679135    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:49.651798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:49.651763    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:49.652844 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:49.652822    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:25:51.050655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.050582    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:25:51.214181 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214144    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-envoy\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214181 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214184    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-certs\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214231    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5478\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-kube-api-access-q5478\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214295    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-podinfo\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214321    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-socket\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214382    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-token\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istiod-ca-cert\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214430    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-data\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214770 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214454    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-credential-socket\") pod \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\" (UID: \"c59abea4-d750-4ffd-b328-c1a6d2800ec8\") "
Jul 28 20:25:51.214770 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214582    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:51.214770 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214678    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:51.214927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214832    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.214927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214853    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.214927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214878    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:51.214927 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.214904    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:25:51.215644 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.215618    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:51.216663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.216640    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:25:51.216867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.216842    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:51.217122 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.217102    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:25:51.217193 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.217168    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-kube-api-access-q5478" (OuterVolumeSpecName: "kube-api-access-q5478") pod "c59abea4-d750-4ffd-b328-c1a6d2800ec8" (UID: "c59abea4-d750-4ffd-b328-c1a6d2800ec8"). InnerVolumeSpecName "kube-api-access-q5478". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315818    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315849    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315859    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315868    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315876    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315886    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c59abea4-d750-4ffd-b328-c1a6d2800ec8-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.315910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.315894    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5478\" (UniqueName: \"kubernetes.io/projected/c59abea4-d750-4ffd-b328-c1a6d2800ec8-kube-api-access-q5478\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:25:51.406783 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.406745    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"]
Jul 28 20:25:51.407255 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.407232    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy"
Jul 28 20:25:51.407383 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.407259    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy"
Jul 28 20:25:51.407383 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.407291    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy"
Jul 28 20:25:51.407383 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.407300    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy"
Jul 28 20:25:51.407550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.407409    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy"
Jul 28 20:25:51.409716 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.409695    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.415021 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.414998    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-711431-serving-cert\""
Jul 28 20:25:51.415164 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.415026    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:25:51.415601 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.415583    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:25:51.416424 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.416395    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:25:51.429980 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.429946    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"]
Jul 28 20:25:51.517693 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.517636    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39467efb-4f82-4245-86ee-51e15f66babe-openshift-service-ca\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.517880 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.517710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59bk5\" (UniqueName: \"kubernetes.io/projected/39467efb-4f82-4245-86ee-51e15f66babe-kube-api-access-59bk5\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.517880 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.517745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39467efb-4f82-4245-86ee-51e15f66babe-maas-api-tls\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.618795 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.618701    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39467efb-4f82-4245-86ee-51e15f66babe-openshift-service-ca\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.618795 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.618756    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59bk5\" (UniqueName: \"kubernetes.io/projected/39467efb-4f82-4245-86ee-51e15f66babe-kube-api-access-59bk5\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.618795 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.618789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39467efb-4f82-4245-86ee-51e15f66babe-maas-api-tls\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.619390 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.619366    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39467efb-4f82-4245-86ee-51e15f66babe-openshift-service-ca\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.621456 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.621436    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39467efb-4f82-4245-86ee-51e15f66babe-maas-api-tls\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.630157 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.630126    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59bk5\" (UniqueName: \"kubernetes.io/projected/39467efb-4f82-4245-86ee-51e15f66babe-kube-api-access-59bk5\") pod \"maas-api-e2e-api-a-711431-868c59bc9f-wdl84\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.663499 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.663464    2578 generic.go:356] "Generic (PLEG): container finished" podID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerID="d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6" exitCode=0
Jul 28 20:25:51.663663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.663511    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"
Jul 28 20:25:51.663663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.663544    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" event={"ID":"c59abea4-d750-4ffd-b328-c1a6d2800ec8","Type":"ContainerDied","Data":"d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6"}
Jul 28 20:25:51.663663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.663597    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k" event={"ID":"c59abea4-d750-4ffd-b328-c1a6d2800ec8","Type":"ContainerDied","Data":"0807bcb6fa46c9cf0e0bcbfb8249b49d7bd4cc778e603f1ab4a429e188d6b0b0"}
Jul 28 20:25:51.663663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.663625    2578 scope.go:117] "RemoveContainer" containerID="d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6"
Jul 28 20:25:51.672912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.672868    2578 scope.go:117] "RemoveContainer" containerID="7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"
Jul 28 20:25:51.682099 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.682076    2578 scope.go:117] "RemoveContainer" containerID="d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6"
Jul 28 20:25:51.682454 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:51.682434    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6\": container with ID starting with d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6 not found: ID does not exist" containerID="d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6"
Jul 28 20:25:51.682547 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.682468    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6"} err="failed to get container status \"d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6\": rpc error: code = NotFound desc = could not find container \"d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6\": container with ID starting with d3db8a14d4f9ab8c0631ede8bef667815b94b87b249449ce0d7a8d07f98559c6 not found: ID does not exist"
Jul 28 20:25:51.682547 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.682495    2578 scope.go:117] "RemoveContainer" containerID="7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"
Jul 28 20:25:51.682754 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:25:51.682733    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558\": container with ID starting with 7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558 not found: ID does not exist" containerID="7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"
Jul 28 20:25:51.682799 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.682762    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558"} err="failed to get container status \"7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558\": rpc error: code = NotFound desc = could not find container \"7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558\": container with ID starting with 7e67984bd3ad0baa190ec0a95df0601c48de55eb021c3cb6060dcc9a33673558 not found: ID does not exist"
Jul 28 20:25:51.689514 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.689482    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"]
Jul 28 20:25:51.695094 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.695063    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-117350-openshift-default-bc5c78698-2q26k"]
Jul 28 20:25:51.721063 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.721022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:51.865736 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.865708    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"]
Jul 28 20:25:51.867420 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:25:51.867383    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39467efb_4f82_4245_86ee_51e15f66babe.slice/crio-5ff3595dbb96574aa1fb334c3787dbf25b45dbdc33302556122767a1a3ec53a8 WatchSource:0}: Error finding container 5ff3595dbb96574aa1fb334c3787dbf25b45dbdc33302556122767a1a3ec53a8: Status 404 returned error can't find the container with id 5ff3595dbb96574aa1fb334c3787dbf25b45dbdc33302556122767a1a3ec53a8
Jul 28 20:25:51.868899 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:51.868851    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:25:52.348513 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.348468    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"]
Jul 28 20:25:52.349071 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.349051    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" containerName="istio-proxy"
Jul 28 20:25:52.352436 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.352414    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"]
Jul 28 20:25:52.352570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.352544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.355034 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.355011    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.359338 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.359315    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-711431-dockercfg-4d494\""
Jul 28 20:25:52.359565 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.359544    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-711431\""
Jul 28 20:25:52.367031 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.367005    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"]
Jul 28 20:25:52.379660 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.379629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"]
Jul 28 20:25:52.425780 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.425746    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mjp2\" (UniqueName: \"kubernetes.io/projected/5c0dc692-c941-4852-ad55-6010fffcd8b7-kube-api-access-6mjp2\") pod \"payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.425968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.425807    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f780897e-9461-4465-9606-8a0aa54ce9d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-711431-74cf598646-vml2r\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.425968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.425849    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0dc692-c941-4852-ad55-6010fffcd8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.425968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.425952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4bf7\" (UniqueName: \"kubernetes.io/projected/f780897e-9461-4465-9606-8a0aa54ce9d4-kube-api-access-m4bf7\") pod \"payload-processing-e2e-api-a-711431-74cf598646-vml2r\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.527286 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.527243    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4bf7\" (UniqueName: \"kubernetes.io/projected/f780897e-9461-4465-9606-8a0aa54ce9d4-kube-api-access-m4bf7\") pod \"payload-processing-e2e-api-a-711431-74cf598646-vml2r\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.527472 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.527307    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mjp2\" (UniqueName: \"kubernetes.io/projected/5c0dc692-c941-4852-ad55-6010fffcd8b7-kube-api-access-6mjp2\") pod \"payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.527472 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.527344    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f780897e-9461-4465-9606-8a0aa54ce9d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-711431-74cf598646-vml2r\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.527472 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.527368    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0dc692-c941-4852-ad55-6010fffcd8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.528326 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.528302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0dc692-c941-4852-ad55-6010fffcd8b7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.528375 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.528302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f780897e-9461-4465-9606-8a0aa54ce9d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-711431-74cf598646-vml2r\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.537664 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.537622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4bf7\" (UniqueName: \"kubernetes.io/projected/f780897e-9461-4465-9606-8a0aa54ce9d4-kube-api-access-m4bf7\") pod \"payload-processing-e2e-api-a-711431-74cf598646-vml2r\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.537939 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.537915    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mjp2\" (UniqueName: \"kubernetes.io/projected/5c0dc692-c941-4852-ad55-6010fffcd8b7-kube-api-access-6mjp2\") pod \"payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.664750 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.664647    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:52.672026 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.671998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:52.672162 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.672117    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84" event={"ID":"39467efb-4f82-4245-86ee-51e15f66babe","Type":"ContainerStarted","Data":"318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6"}
Jul 28 20:25:52.672231 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.672169    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84" event={"ID":"39467efb-4f82-4245-86ee-51e15f66babe","Type":"ContainerStarted","Data":"5ff3595dbb96574aa1fb334c3787dbf25b45dbdc33302556122767a1a3ec53a8"}
Jul 28 20:25:52.672315 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.672239    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:25:52.703246 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.703183    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84" podStartSLOduration=1.17015377 podStartE2EDuration="1.703164015s" totalImagesPullingTime="533.010245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:25:52.701392407 +0000 UTC m=+2355.601609250" watchObservedRunningTime="2026-07-28 20:25:52.703164015 +0000 UTC m=+2355.603380856"
Jul 28 20:25:52.847591 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.847556    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"]
Jul 28 20:25:52.849573 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:25:52.849510    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf780897e_9461_4465_9606_8a0aa54ce9d4.slice/crio-c4f380342e397d1e5f99a62bdbdf10c705e52f41fcea20b27b87b50bb38940c3 WatchSource:0}: Error finding container c4f380342e397d1e5f99a62bdbdf10c705e52f41fcea20b27b87b50bb38940c3: Status 404 returned error can't find the container with id c4f380342e397d1e5f99a62bdbdf10c705e52f41fcea20b27b87b50bb38940c3
Jul 28 20:25:52.872545 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:52.872450    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"]
Jul 28 20:25:52.874817 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:25:52.874786    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0dc692_c941_4852_ad55_6010fffcd8b7.slice/crio-ff939c88355f952d0c1268aa542372e2cf573bb1209040be41be992d5bd4f5c2 WatchSource:0}: Error finding container ff939c88355f952d0c1268aa542372e2cf573bb1209040be41be992d5bd4f5c2: Status 404 returned error can't find the container with id ff939c88355f952d0c1268aa542372e2cf573bb1209040be41be992d5bd4f5c2
Jul 28 20:25:53.657967 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.657930    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c59abea4-d750-4ffd-b328-c1a6d2800ec8" path="/var/lib/kubelet/pods/c59abea4-d750-4ffd-b328-c1a6d2800ec8/volumes"
Jul 28 20:25:53.679511 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.679471    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88" event={"ID":"5c0dc692-c941-4852-ad55-6010fffcd8b7","Type":"ContainerStarted","Data":"6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b"}
Jul 28 20:25:53.679703 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.679540    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88" event={"ID":"5c0dc692-c941-4852-ad55-6010fffcd8b7","Type":"ContainerStarted","Data":"ff939c88355f952d0c1268aa542372e2cf573bb1209040be41be992d5bd4f5c2"}
Jul 28 20:25:53.679703 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.679656    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:25:53.680983 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.680957    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r" event={"ID":"f780897e-9461-4465-9606-8a0aa54ce9d4","Type":"ContainerStarted","Data":"454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49"}
Jul 28 20:25:53.681130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.680996    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r" event={"ID":"f780897e-9461-4465-9606-8a0aa54ce9d4","Type":"ContainerStarted","Data":"c4f380342e397d1e5f99a62bdbdf10c705e52f41fcea20b27b87b50bb38940c3"}
Jul 28 20:25:53.681130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.681015    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:25:53.703019 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.702962    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88" podStartSLOduration=1.702944314 podStartE2EDuration="1.702944314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:25:53.70141044 +0000 UTC m=+2356.601627310" watchObservedRunningTime="2026-07-28 20:25:53.702944314 +0000 UTC m=+2356.603161308"
Jul 28 20:25:53.723250 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:53.723184    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r" podStartSLOduration=1.723160679 podStartE2EDuration="1.723160679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:25:53.72037413 +0000 UTC m=+2356.620590973" watchObservedRunningTime="2026-07-28 20:25:53.723160679 +0000 UTC m=+2356.623377523"
Jul 28 20:25:56.800832 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:56.800796    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:25:58.686456 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:25:58.686418    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:26:04.653359 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.653232    2578 patch_prober.go:28] interesting pod/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 20:26:04.653359 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.653332    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 20:26:04.688567 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.688467    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:26:04.688567 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.688548    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:26:04.749579 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.749533    2578 generic.go:356] "Generic (PLEG): container finished" podID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerID="6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92" exitCode=0
Jul 28 20:26:04.749756 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.749601    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" event={"ID":"c52f5574-851d-4692-84a8-a0b9ab767b49","Type":"ContainerDied","Data":"6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"}
Jul 28 20:26:04.749998 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.749981    2578 scope.go:117] "RemoveContainer" containerID="6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"
Jul 28 20:26:04.751144 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.751111    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:26:04.751249 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.751223    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:26:04.751339 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:04.751296    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:26:05.762952 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:05.762898    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" event={"ID":"c52f5574-851d-4692-84a8-a0b9ab767b49","Type":"ContainerStarted","Data":"08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23"}
Jul 28 20:26:06.674652 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:06.674602    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:26:06.674913 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:06.674886    2578 patch_prober.go:28] interesting pod/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 20:26:06.674999 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:06.674944    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 20:26:07.674792 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:07.674747    2578 patch_prober.go:28] interesting pod/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 20:26:07.675236 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:07.674821    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 20:26:08.674806 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:08.674764    2578 patch_prober.go:28] interesting pod/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 20:26:08.675243 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:08.674822    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 20:26:09.674194 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:09.674149    2578 patch_prober.go:28] interesting pod/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 20:26:09.674415 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:09.674224    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 20:26:10.678622 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:10.678582    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:26:10.679036 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:10.678846    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:26:10.679615 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:10.679594    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:26:21.211473 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.211434    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"]
Jul 28 20:26:21.213940 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.213922    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.218598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.218568    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f632f1-serving-cert\""
Jul 28 20:26:21.232569 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.232535    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"]
Jul 28 20:26:21.288465 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.288419    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c84703-2327-4528-9f53-4639b5eae034-maas-api-tls\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.288651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.288478    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssw2d\" (UniqueName: \"kubernetes.io/projected/23c84703-2327-4528-9f53-4639b5eae034-kube-api-access-ssw2d\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.288651 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.288590    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c84703-2327-4528-9f53-4639b5eae034-openshift-service-ca\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.390068 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.390022    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c84703-2327-4528-9f53-4639b5eae034-maas-api-tls\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.390320 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.390076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssw2d\" (UniqueName: \"kubernetes.io/projected/23c84703-2327-4528-9f53-4639b5eae034-kube-api-access-ssw2d\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.390320 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.390163    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c84703-2327-4528-9f53-4639b5eae034-openshift-service-ca\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.390880 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.390854    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c84703-2327-4528-9f53-4639b5eae034-openshift-service-ca\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.392704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.392681    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c84703-2327-4528-9f53-4639b5eae034-maas-api-tls\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.404837 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.404805    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssw2d\" (UniqueName: \"kubernetes.io/projected/23c84703-2327-4528-9f53-4639b5eae034-kube-api-access-ssw2d\") pod \"maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.524304 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.524244    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:21.679795 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.679766    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"]
Jul 28 20:26:21.681938 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:26:21.681901    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c84703_2327_4528_9f53_4639b5eae034.slice/crio-1ed46e1166a0cf03bf6b67954930a484ffcb398293717e60b73de397f0bd22a0 WatchSource:0}: Error finding container 1ed46e1166a0cf03bf6b67954930a484ffcb398293717e60b73de397f0bd22a0: Status 404 returned error can't find the container with id 1ed46e1166a0cf03bf6b67954930a484ffcb398293717e60b73de397f0bd22a0
Jul 28 20:26:21.850715 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.850613    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" event={"ID":"23c84703-2327-4528-9f53-4639b5eae034","Type":"ContainerStarted","Data":"1ed46e1166a0cf03bf6b67954930a484ffcb398293717e60b73de397f0bd22a0"}
Jul 28 20:26:21.988492 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.988456    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"]
Jul 28 20:26:21.991519 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.991493    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:21.994536 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:21.994511    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f632f1\""
Jul 28 20:26:22.007631 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.007597    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"]
Jul 28 20:26:22.052666 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.052627    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"]
Jul 28 20:26:22.055028 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.055008    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.058293 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.058255    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f632f1-dockercfg-cvkcd\""
Jul 28 20:26:22.067515 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.067484    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"]
Jul 28 20:26:22.096090 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.096051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86100670-1faf-47bc-835b-2347e17eae45-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") " pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.096311 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.096098    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdfds\" (UniqueName: \"kubernetes.io/projected/1caddf2d-246e-41f0-a97a-48390d5da35d-kube-api-access-cdfds\") pod \"payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.096311 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.096174    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caddf2d-246e-41f0-a97a-48390d5da35d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.096311 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.096220    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27vkw\" (UniqueName: \"kubernetes.io/projected/86100670-1faf-47bc-835b-2347e17eae45-kube-api-access-27vkw\") pod \"payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") " pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.197696 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.197593    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caddf2d-246e-41f0-a97a-48390d5da35d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.197696 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.197651    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27vkw\" (UniqueName: \"kubernetes.io/projected/86100670-1faf-47bc-835b-2347e17eae45-kube-api-access-27vkw\") pod \"payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") " pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.197909 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.197703    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86100670-1faf-47bc-835b-2347e17eae45-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") " pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.197909 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.197723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdfds\" (UniqueName: \"kubernetes.io/projected/1caddf2d-246e-41f0-a97a-48390d5da35d-kube-api-access-cdfds\") pod \"payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.198446 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.198427    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86100670-1faf-47bc-835b-2347e17eae45-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") " pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.198494 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.198460    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caddf2d-246e-41f0-a97a-48390d5da35d-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.206984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.206944    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27vkw\" (UniqueName: \"kubernetes.io/projected/86100670-1faf-47bc-835b-2347e17eae45-kube-api-access-27vkw\") pod \"payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") " pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.206984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.206978    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdfds\" (UniqueName: \"kubernetes.io/projected/1caddf2d-246e-41f0-a97a-48390d5da35d-kube-api-access-cdfds\") pod \"payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.302950 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.302906    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.365676 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.365638    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.461874 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.461831    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"]
Jul 28 20:26:22.467234 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:26:22.467206    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86100670_1faf_47bc_835b_2347e17eae45.slice/crio-f11bf4c5ff8f3e4b442c4a3db93a8a9ce739a8c0690a485e97af7789fcb61b40 WatchSource:0}: Error finding container f11bf4c5ff8f3e4b442c4a3db93a8a9ce739a8c0690a485e97af7789fcb61b40: Status 404 returned error can't find the container with id f11bf4c5ff8f3e4b442c4a3db93a8a9ce739a8c0690a485e97af7789fcb61b40
Jul 28 20:26:22.540931 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.540901    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"]
Jul 28 20:26:22.542437 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:26:22.542379    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1caddf2d_246e_41f0_a97a_48390d5da35d.slice/crio-3c0404d649b2dad4cf2eeb8744dc7d150205d4459c525080d9a71e2cfafe0e69 WatchSource:0}: Error finding container 3c0404d649b2dad4cf2eeb8744dc7d150205d4459c525080d9a71e2cfafe0e69: Status 404 returned error can't find the container with id 3c0404d649b2dad4cf2eeb8744dc7d150205d4459c525080d9a71e2cfafe0e69
Jul 28 20:26:22.857465 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.857420    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" event={"ID":"86100670-1faf-47bc-835b-2347e17eae45","Type":"ContainerStarted","Data":"d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3"}
Jul 28 20:26:22.857711 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.857485    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" event={"ID":"86100670-1faf-47bc-835b-2347e17eae45","Type":"ContainerStarted","Data":"f11bf4c5ff8f3e4b442c4a3db93a8a9ce739a8c0690a485e97af7789fcb61b40"}
Jul 28 20:26:22.857711 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.857598    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:22.859321 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.859290    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" event={"ID":"1caddf2d-246e-41f0-a97a-48390d5da35d","Type":"ContainerStarted","Data":"94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c"}
Jul 28 20:26:22.859462 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.859338    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" event={"ID":"1caddf2d-246e-41f0-a97a-48390d5da35d","Type":"ContainerStarted","Data":"3c0404d649b2dad4cf2eeb8744dc7d150205d4459c525080d9a71e2cfafe0e69"}
Jul 28 20:26:22.859462 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.859369    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:22.861005 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.860975    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" event={"ID":"23c84703-2327-4528-9f53-4639b5eae034","Type":"ContainerStarted","Data":"196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682"}
Jul 28 20:26:22.861153 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.861107    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:22.877071 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.877015    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" podStartSLOduration=1.876997658 podStartE2EDuration="1.876997658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:26:22.875418986 +0000 UTC m=+2385.775635829" watchObservedRunningTime="2026-07-28 20:26:22.876997658 +0000 UTC m=+2385.777214499"
Jul 28 20:26:22.891874 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.891816    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" podStartSLOduration=0.89180084 podStartE2EDuration="891.80084ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:26:22.891005625 +0000 UTC m=+2385.791222482" watchObservedRunningTime="2026-07-28 20:26:22.89180084 +0000 UTC m=+2385.792017727"
Jul 28 20:26:22.912687 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:22.912634    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" podStartSLOduration=1.216827888 podStartE2EDuration="1.912617565s" totalImagesPullingTime="695.789677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:26:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:26:22.91030854 +0000 UTC m=+2385.810525383" watchObservedRunningTime="2026-07-28 20:26:22.912617565 +0000 UTC m=+2385.812834435"
Jul 28 20:26:25.520791 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:25.520741    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:26:28.872839 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:28.872807    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:26:33.868650 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:33.868610    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:26:33.869092 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:33.868983    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:26:37.739518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:37.739485    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:26:37.744557 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:37.744527    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:26:40.138528 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:26:40.138488    2578 scope.go:117] "RemoveContainer" containerID="7a417de0c043dbd3c2670e5d10a714cfb3894f0c29d1a4a34c65b7742aa14a57"
Jul 28 20:27:10.093424 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.093383    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"]
Jul 28 20:27:10.094079 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.093643    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84" podUID="39467efb-4f82-4245-86ee-51e15f66babe" containerName="maas-api" containerID="cri-o://318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6" gracePeriod=30
Jul 28 20:27:10.156365 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.156327    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"]
Jul 28 20:27:10.156611 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.156578    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r" podUID="f780897e-9461-4465-9606-8a0aa54ce9d4" containerName="payload-processing" containerID="cri-o://454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49" gracePeriod=30
Jul 28 20:27:10.170867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.170834    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"]
Jul 28 20:27:10.171049 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.171026    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88" podUID="5c0dc692-c941-4852-ad55-6010fffcd8b7" containerName="payload-pre-processing" containerID="cri-o://6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b" gracePeriod=30
Jul 28 20:27:10.467305 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.467243    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:27:10.471211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.471181    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:27:10.487109 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.487078    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:27:10.528172 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528139    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0dc692-c941-4852-ad55-6010fffcd8b7-plugins-config-volume\") pod \"5c0dc692-c941-4852-ad55-6010fffcd8b7\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") "
Jul 28 20:27:10.528400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528181    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59bk5\" (UniqueName: \"kubernetes.io/projected/39467efb-4f82-4245-86ee-51e15f66babe-kube-api-access-59bk5\") pod \"39467efb-4f82-4245-86ee-51e15f66babe\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") "
Jul 28 20:27:10.528400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528206    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6mjp2\" (UniqueName: \"kubernetes.io/projected/5c0dc692-c941-4852-ad55-6010fffcd8b7-kube-api-access-6mjp2\") pod \"5c0dc692-c941-4852-ad55-6010fffcd8b7\" (UID: \"5c0dc692-c941-4852-ad55-6010fffcd8b7\") "
Jul 28 20:27:10.528400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528241    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39467efb-4f82-4245-86ee-51e15f66babe-maas-api-tls\") pod \"39467efb-4f82-4245-86ee-51e15f66babe\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") "
Jul 28 20:27:10.528400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528297    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4bf7\" (UniqueName: \"kubernetes.io/projected/f780897e-9461-4465-9606-8a0aa54ce9d4-kube-api-access-m4bf7\") pod \"f780897e-9461-4465-9606-8a0aa54ce9d4\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") "
Jul 28 20:27:10.528400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f780897e-9461-4465-9606-8a0aa54ce9d4-plugins-config-volume\") pod \"f780897e-9461-4465-9606-8a0aa54ce9d4\" (UID: \"f780897e-9461-4465-9606-8a0aa54ce9d4\") "
Jul 28 20:27:10.528400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528377    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39467efb-4f82-4245-86ee-51e15f66babe-openshift-service-ca\") pod \"39467efb-4f82-4245-86ee-51e15f66babe\" (UID: \"39467efb-4f82-4245-86ee-51e15f66babe\") "
Jul 28 20:27:10.528746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528603    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0dc692-c941-4852-ad55-6010fffcd8b7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c0dc692-c941-4852-ad55-6010fffcd8b7" (UID: "5c0dc692-c941-4852-ad55-6010fffcd8b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:10.528746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528718    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c0dc692-c941-4852-ad55-6010fffcd8b7-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:10.529228 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.528959    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39467efb-4f82-4245-86ee-51e15f66babe-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "39467efb-4f82-4245-86ee-51e15f66babe" (UID: "39467efb-4f82-4245-86ee-51e15f66babe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:10.529400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.529369    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f780897e-9461-4465-9606-8a0aa54ce9d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f780897e-9461-4465-9606-8a0aa54ce9d4" (UID: "f780897e-9461-4465-9606-8a0aa54ce9d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:10.530788 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.530764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39467efb-4f82-4245-86ee-51e15f66babe-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "39467efb-4f82-4245-86ee-51e15f66babe" (UID: "39467efb-4f82-4245-86ee-51e15f66babe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:27:10.530883 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.530791    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f780897e-9461-4465-9606-8a0aa54ce9d4-kube-api-access-m4bf7" (OuterVolumeSpecName: "kube-api-access-m4bf7") pod "f780897e-9461-4465-9606-8a0aa54ce9d4" (UID: "f780897e-9461-4465-9606-8a0aa54ce9d4"). InnerVolumeSpecName "kube-api-access-m4bf7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:10.530883 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.530803    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39467efb-4f82-4245-86ee-51e15f66babe-kube-api-access-59bk5" (OuterVolumeSpecName: "kube-api-access-59bk5") pod "39467efb-4f82-4245-86ee-51e15f66babe" (UID: "39467efb-4f82-4245-86ee-51e15f66babe"). InnerVolumeSpecName "kube-api-access-59bk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:10.531004 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.530977    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0dc692-c941-4852-ad55-6010fffcd8b7-kube-api-access-6mjp2" (OuterVolumeSpecName: "kube-api-access-6mjp2") pod "5c0dc692-c941-4852-ad55-6010fffcd8b7" (UID: "5c0dc692-c941-4852-ad55-6010fffcd8b7"). InnerVolumeSpecName "kube-api-access-6mjp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:10.629534 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.629437    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59bk5\" (UniqueName: \"kubernetes.io/projected/39467efb-4f82-4245-86ee-51e15f66babe-kube-api-access-59bk5\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:10.629534 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.629471    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mjp2\" (UniqueName: \"kubernetes.io/projected/5c0dc692-c941-4852-ad55-6010fffcd8b7-kube-api-access-6mjp2\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:10.629534 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.629482    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/39467efb-4f82-4245-86ee-51e15f66babe-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:10.629534 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.629491    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4bf7\" (UniqueName: \"kubernetes.io/projected/f780897e-9461-4465-9606-8a0aa54ce9d4-kube-api-access-m4bf7\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:10.629534 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.629499    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f780897e-9461-4465-9606-8a0aa54ce9d4-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:10.629534 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:10.629508    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/39467efb-4f82-4245-86ee-51e15f66babe-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:11.123976 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.123939    2578 generic.go:356] "Generic (PLEG): container finished" podID="5c0dc692-c941-4852-ad55-6010fffcd8b7" containerID="6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b" exitCode=0
Jul 28 20:27:11.123976 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.123978    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"
Jul 28 20:27:11.124489 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.124027    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88" event={"ID":"5c0dc692-c941-4852-ad55-6010fffcd8b7","Type":"ContainerDied","Data":"6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b"}
Jul 28 20:27:11.124489 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.124079    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88" event={"ID":"5c0dc692-c941-4852-ad55-6010fffcd8b7","Type":"ContainerDied","Data":"ff939c88355f952d0c1268aa542372e2cf573bb1209040be41be992d5bd4f5c2"}
Jul 28 20:27:11.124489 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.124108    2578 scope.go:117] "RemoveContainer" containerID="6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b"
Jul 28 20:27:11.125507 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.125487    2578 generic.go:356] "Generic (PLEG): container finished" podID="39467efb-4f82-4245-86ee-51e15f66babe" containerID="318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6" exitCode=0
Jul 28 20:27:11.125577 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.125526    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"
Jul 28 20:27:11.125577 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.125556    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84" event={"ID":"39467efb-4f82-4245-86ee-51e15f66babe","Type":"ContainerDied","Data":"318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6"}
Jul 28 20:27:11.125689 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.125602    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84" event={"ID":"39467efb-4f82-4245-86ee-51e15f66babe","Type":"ContainerDied","Data":"5ff3595dbb96574aa1fb334c3787dbf25b45dbdc33302556122767a1a3ec53a8"}
Jul 28 20:27:11.126965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.126943    2578 generic.go:356] "Generic (PLEG): container finished" podID="f780897e-9461-4465-9606-8a0aa54ce9d4" containerID="454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49" exitCode=0
Jul 28 20:27:11.127116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.126990    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"
Jul 28 20:27:11.127116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.127023    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r" event={"ID":"f780897e-9461-4465-9606-8a0aa54ce9d4","Type":"ContainerDied","Data":"454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49"}
Jul 28 20:27:11.127116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.127074    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r" event={"ID":"f780897e-9461-4465-9606-8a0aa54ce9d4","Type":"ContainerDied","Data":"c4f380342e397d1e5f99a62bdbdf10c705e52f41fcea20b27b87b50bb38940c3"}
Jul 28 20:27:11.136182 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.136156    2578 scope.go:117] "RemoveContainer" containerID="6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b"
Jul 28 20:27:11.136563 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:11.136531    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b\": container with ID starting with 6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b not found: ID does not exist" containerID="6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b"
Jul 28 20:27:11.136654 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.136574    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b"} err="failed to get container status \"6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b\": rpc error: code = NotFound desc = could not find container \"6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b\": container with ID starting with 6507d1db8e480f813b6a1c41de767dbfb92477e7e98d94f43dd3e96bf423358b not found: ID does not exist"
Jul 28 20:27:11.136654 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.136594    2578 scope.go:117] "RemoveContainer" containerID="318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6"
Jul 28 20:27:11.145469 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.145438    2578 scope.go:117] "RemoveContainer" containerID="318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6"
Jul 28 20:27:11.145772 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:11.145749    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6\": container with ID starting with 318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6 not found: ID does not exist" containerID="318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6"
Jul 28 20:27:11.145825 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.145782    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6"} err="failed to get container status \"318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6\": rpc error: code = NotFound desc = could not find container \"318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6\": container with ID starting with 318ee24f5025426b670049a3f97e7b23192f669ca169d8d0dc6f2e821cb8b3b6 not found: ID does not exist"
Jul 28 20:27:11.145825 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.145801    2578 scope.go:117] "RemoveContainer" containerID="454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49"
Jul 28 20:27:11.154254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.154234    2578 scope.go:117] "RemoveContainer" containerID="454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49"
Jul 28 20:27:11.154621 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:11.154599    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49\": container with ID starting with 454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49 not found: ID does not exist" containerID="454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49"
Jul 28 20:27:11.154664 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.154632    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49"} err="failed to get container status \"454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49\": rpc error: code = NotFound desc = could not find container \"454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49\": container with ID starting with 454010fb3703fdea7f1a65a9e731a85f12037f2f327386fc318d593d5a4c9b49 not found: ID does not exist"
Jul 28 20:27:11.161817 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.161785    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"]
Jul 28 20:27:11.167257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.167225    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-711431-74cf598646-vml2r"]
Jul 28 20:27:11.185241 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.185207    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"]
Jul 28 20:27:11.188251 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.188222    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-711431-5ddb54f4d8-cqf88"]
Jul 28 20:27:11.206062 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.206019    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"]
Jul 28 20:27:11.222497 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.222469    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-711431-868c59bc9f-wdl84"]
Jul 28 20:27:11.650820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.650780    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39467efb-4f82-4245-86ee-51e15f66babe" path="/var/lib/kubelet/pods/39467efb-4f82-4245-86ee-51e15f66babe/volumes"
Jul 28 20:27:11.651158 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.651144    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c0dc692-c941-4852-ad55-6010fffcd8b7" path="/var/lib/kubelet/pods/5c0dc692-c941-4852-ad55-6010fffcd8b7/volumes"
Jul 28 20:27:11.651511 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:11.651498    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f780897e-9461-4465-9606-8a0aa54ce9d4" path="/var/lib/kubelet/pods/f780897e-9461-4465-9606-8a0aa54ce9d4/volumes"
Jul 28 20:27:16.403116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:16.403078    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:27:17.776907 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:17.776865    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"]
Jul 28 20:27:17.777416 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:17.777074    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy" containerID="cri-o://08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23" gracePeriod=30
Jul 28 20:27:23.027372 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.027341    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:27:23.141055 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.140956    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-data\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141055 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141008    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-socket\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141055 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141025    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-envoy\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141055 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141045    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-credential-socket\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141071    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-token\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141143    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f5574-851d-4692-84a8-a0b9ab767b49-istiod-ca-cert\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141179    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9x7m\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-kube-api-access-f9x7m\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141211    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-podinfo\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141250    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-certs\") pod \"c52f5574-851d-4692-84a8-a0b9ab767b49\" (UID: \"c52f5574-851d-4692-84a8-a0b9ab767b49\") "
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141300    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:27:23.141453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141383    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:27:23.141829 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141590    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.141829 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141612    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.141829 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141592    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:27:23.141829 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141634    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52f5574-851d-4692-84a8-a0b9ab767b49-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:23.142017 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.141861    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-data" (OuterVolumeSpecName: "istio-data") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:27:23.143566 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.143543    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:27:23.143827 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.143795    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-kube-api-access-f9x7m" (OuterVolumeSpecName: "kube-api-access-f9x7m") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "kube-api-access-f9x7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:23.144119 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.144099    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:27:23.144168 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.144137    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-token" (OuterVolumeSpecName: "istio-token") pod "c52f5574-851d-4692-84a8-a0b9ab767b49" (UID: "c52f5574-851d-4692-84a8-a0b9ab767b49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:23.195474 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.195431    2578 generic.go:356] "Generic (PLEG): container finished" podID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerID="08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23" exitCode=0
Jul 28 20:27:23.195678 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.195479    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"
Jul 28 20:27:23.195678 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.195520    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" event={"ID":"c52f5574-851d-4692-84a8-a0b9ab767b49","Type":"ContainerDied","Data":"08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23"}
Jul 28 20:27:23.195678 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.195614    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v" event={"ID":"c52f5574-851d-4692-84a8-a0b9ab767b49","Type":"ContainerDied","Data":"9ba22e6151d3ec8dd6bcba9b7a72e5e9d0031bdae723a32799c925fb1a6f9cf9"}
Jul 28 20:27:23.195678 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.195646    2578 scope.go:117] "RemoveContainer" containerID="08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23"
Jul 28 20:27:23.207460 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.207321    2578 scope.go:117] "RemoveContainer" containerID="6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"
Jul 28 20:27:23.217281 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.217248    2578 scope.go:117] "RemoveContainer" containerID="08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23"
Jul 28 20:27:23.217605 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:23.217583    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23\": container with ID starting with 08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23 not found: ID does not exist" containerID="08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23"
Jul 28 20:27:23.217659 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.217614    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23"} err="failed to get container status \"08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23\": rpc error: code = NotFound desc = could not find container \"08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23\": container with ID starting with 08a3261e2f3d4b27321676d5bd42b6e0f127823f0a3e4526e4eaffd0af4caa23 not found: ID does not exist"
Jul 28 20:27:23.217659 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.217632    2578 scope.go:117] "RemoveContainer" containerID="6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"
Jul 28 20:27:23.217879 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:23.217856    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92\": container with ID starting with 6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92 not found: ID does not exist" containerID="6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"
Jul 28 20:27:23.217942 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.217890    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92"} err="failed to get container status \"6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92\": rpc error: code = NotFound desc = could not find container \"6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92\": container with ID starting with 6b5d1255cf8ec6fc6a4ace8d1aa2f0fbda9cce4f79b24473c47d49540b686b92 not found: ID does not exist"
Jul 28 20:27:23.225912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.225875    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"]
Jul 28 20:27:23.227964 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.227934    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-711431-openshift-default-6bc756c986-w6d6v"]
Jul 28 20:27:23.242734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242700    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c52f5574-851d-4692-84a8-a0b9ab767b49-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.242734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242732    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9x7m\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-kube-api-access-f9x7m\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.242866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242742    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.242866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242752    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.242866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242763    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.242866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242771    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.242866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.242778    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c52f5574-851d-4692-84a8-a0b9ab767b49-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:23.651005 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:23.650969    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" path="/var/lib/kubelet/pods/c52f5574-851d-4692-84a8-a0b9ab767b49/volumes"
Jul 28 20:27:33.717256 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.717208    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"]
Jul 28 20:27:33.717765 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.717504    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" podUID="23c84703-2327-4528-9f53-4639b5eae034" containerName="maas-api" containerID="cri-o://196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682" gracePeriod=30
Jul 28 20:27:33.774310 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.774261    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"]
Jul 28 20:27:33.774674 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.774490    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" podUID="86100670-1faf-47bc-835b-2347e17eae45" containerName="payload-processing" containerID="cri-o://d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3" gracePeriod=30
Jul 28 20:27:33.789806 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.789773    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"]
Jul 28 20:27:33.790059 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.790018    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" podUID="1caddf2d-246e-41f0-a97a-48390d5da35d" containerName="payload-pre-processing" containerID="cri-o://94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c" gracePeriod=30
Jul 28 20:27:33.867996 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.867952    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" podUID="86100670-1faf-47bc-835b-2347e17eae45" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.75:9004: connect: connection refused"
Jul 28 20:27:33.867996 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.867974    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" podUID="23c84703-2327-4528-9f53-4639b5eae034" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.74:8443/health\": dial tcp 10.132.0.74:8443: connect: connection refused"
Jul 28 20:27:33.868225 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:33.867972    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" podUID="1caddf2d-246e-41f0-a97a-48390d5da35d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.76:9004: connect: connection refused"
Jul 28 20:27:34.042128 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.042099    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:27:34.083067 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.083033    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:27:34.086459 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.086437    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:27:34.135737 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.135700    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c84703-2327-4528-9f53-4639b5eae034-maas-api-tls\") pod \"23c84703-2327-4528-9f53-4639b5eae034\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") "
Jul 28 20:27:34.135919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.135784    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c84703-2327-4528-9f53-4639b5eae034-openshift-service-ca\") pod \"23c84703-2327-4528-9f53-4639b5eae034\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") "
Jul 28 20:27:34.136007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.135936    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssw2d\" (UniqueName: \"kubernetes.io/projected/23c84703-2327-4528-9f53-4639b5eae034-kube-api-access-ssw2d\") pod \"23c84703-2327-4528-9f53-4639b5eae034\" (UID: \"23c84703-2327-4528-9f53-4639b5eae034\") "
Jul 28 20:27:34.136161 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.136136    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c84703-2327-4528-9f53-4639b5eae034-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "23c84703-2327-4528-9f53-4639b5eae034" (UID: "23c84703-2327-4528-9f53-4639b5eae034"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:34.136294 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.136250    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c84703-2327-4528-9f53-4639b5eae034-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:34.138086 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.138051    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c84703-2327-4528-9f53-4639b5eae034-kube-api-access-ssw2d" (OuterVolumeSpecName: "kube-api-access-ssw2d") pod "23c84703-2327-4528-9f53-4639b5eae034" (UID: "23c84703-2327-4528-9f53-4639b5eae034"). InnerVolumeSpecName "kube-api-access-ssw2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:34.138086 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.138060    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c84703-2327-4528-9f53-4639b5eae034-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "23c84703-2327-4528-9f53-4639b5eae034" (UID: "23c84703-2327-4528-9f53-4639b5eae034"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:27:34.237400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237304    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86100670-1faf-47bc-835b-2347e17eae45-plugins-config-volume\") pod \"86100670-1faf-47bc-835b-2347e17eae45\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") "
Jul 28 20:27:34.237400 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237371    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdfds\" (UniqueName: \"kubernetes.io/projected/1caddf2d-246e-41f0-a97a-48390d5da35d-kube-api-access-cdfds\") pod \"1caddf2d-246e-41f0-a97a-48390d5da35d\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") "
Jul 28 20:27:34.237592 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237543    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27vkw\" (UniqueName: \"kubernetes.io/projected/86100670-1faf-47bc-835b-2347e17eae45-kube-api-access-27vkw\") pod \"86100670-1faf-47bc-835b-2347e17eae45\" (UID: \"86100670-1faf-47bc-835b-2347e17eae45\") "
Jul 28 20:27:34.237636 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237612    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caddf2d-246e-41f0-a97a-48390d5da35d-plugins-config-volume\") pod \"1caddf2d-246e-41f0-a97a-48390d5da35d\" (UID: \"1caddf2d-246e-41f0-a97a-48390d5da35d\") "
Jul 28 20:27:34.237735 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237709    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86100670-1faf-47bc-835b-2347e17eae45-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "86100670-1faf-47bc-835b-2347e17eae45" (UID: "86100670-1faf-47bc-835b-2347e17eae45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:34.237945 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237925    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssw2d\" (UniqueName: \"kubernetes.io/projected/23c84703-2327-4528-9f53-4639b5eae034-kube-api-access-ssw2d\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:34.237945 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237945    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86100670-1faf-47bc-835b-2347e17eae45-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:34.238116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.237958    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/23c84703-2327-4528-9f53-4639b5eae034-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:34.238116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.238000    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1caddf2d-246e-41f0-a97a-48390d5da35d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1caddf2d-246e-41f0-a97a-48390d5da35d" (UID: "1caddf2d-246e-41f0-a97a-48390d5da35d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:27:34.239563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.239541    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1caddf2d-246e-41f0-a97a-48390d5da35d-kube-api-access-cdfds" (OuterVolumeSpecName: "kube-api-access-cdfds") pod "1caddf2d-246e-41f0-a97a-48390d5da35d" (UID: "1caddf2d-246e-41f0-a97a-48390d5da35d"). InnerVolumeSpecName "kube-api-access-cdfds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:34.239793 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.239772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86100670-1faf-47bc-835b-2347e17eae45-kube-api-access-27vkw" (OuterVolumeSpecName: "kube-api-access-27vkw") pod "86100670-1faf-47bc-835b-2347e17eae45" (UID: "86100670-1faf-47bc-835b-2347e17eae45"). InnerVolumeSpecName "kube-api-access-27vkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:27:34.255641 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.255608    2578 generic.go:356] "Generic (PLEG): container finished" podID="23c84703-2327-4528-9f53-4639b5eae034" containerID="196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682" exitCode=0
Jul 28 20:27:34.255823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.255660    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"
Jul 28 20:27:34.255823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.255688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" event={"ID":"23c84703-2327-4528-9f53-4639b5eae034","Type":"ContainerDied","Data":"196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682"}
Jul 28 20:27:34.255823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.255744    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7" event={"ID":"23c84703-2327-4528-9f53-4639b5eae034","Type":"ContainerDied","Data":"1ed46e1166a0cf03bf6b67954930a484ffcb398293717e60b73de397f0bd22a0"}
Jul 28 20:27:34.255823 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.255768    2578 scope.go:117] "RemoveContainer" containerID="196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682"
Jul 28 20:27:34.257223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.257199    2578 generic.go:356] "Generic (PLEG): container finished" podID="86100670-1faf-47bc-835b-2347e17eae45" containerID="d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3" exitCode=0
Jul 28 20:27:34.257363 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.257236    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"
Jul 28 20:27:34.257363 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.257299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" event={"ID":"86100670-1faf-47bc-835b-2347e17eae45","Type":"ContainerDied","Data":"d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3"}
Jul 28 20:27:34.257633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.257606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d" event={"ID":"86100670-1faf-47bc-835b-2347e17eae45","Type":"ContainerDied","Data":"f11bf4c5ff8f3e4b442c4a3db93a8a9ce739a8c0690a485e97af7789fcb61b40"}
Jul 28 20:27:34.258866 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.258844    2578 generic.go:356] "Generic (PLEG): container finished" podID="1caddf2d-246e-41f0-a97a-48390d5da35d" containerID="94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c" exitCode=0
Jul 28 20:27:34.258968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.258867    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" event={"ID":"1caddf2d-246e-41f0-a97a-48390d5da35d","Type":"ContainerDied","Data":"94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c"}
Jul 28 20:27:34.258968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.258900    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"
Jul 28 20:27:34.258968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.258915    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8" event={"ID":"1caddf2d-246e-41f0-a97a-48390d5da35d","Type":"ContainerDied","Data":"3c0404d649b2dad4cf2eeb8744dc7d150205d4459c525080d9a71e2cfafe0e69"}
Jul 28 20:27:34.270770 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.270745    2578 scope.go:117] "RemoveContainer" containerID="196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682"
Jul 28 20:27:34.271310 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:34.271239    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682\": container with ID starting with 196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682 not found: ID does not exist" containerID="196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682"
Jul 28 20:27:34.271427 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.271305    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682"} err="failed to get container status \"196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682\": rpc error: code = NotFound desc = could not find container \"196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682\": container with ID starting with 196e65d1c1e1e84d3e8a27878ec22c162b8bbcb996bef77c466ed9b599c8c682 not found: ID does not exist"
Jul 28 20:27:34.271427 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.271331    2578 scope.go:117] "RemoveContainer" containerID="d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3"
Jul 28 20:27:34.280884 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.280865    2578 scope.go:117] "RemoveContainer" containerID="d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3"
Jul 28 20:27:34.281179 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:34.281159    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3\": container with ID starting with d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3 not found: ID does not exist" containerID="d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3"
Jul 28 20:27:34.281221 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.281189    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3"} err="failed to get container status \"d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3\": rpc error: code = NotFound desc = could not find container \"d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3\": container with ID starting with d4f4757504ad5c9899fdc90a513bb82f0197cd6b7f78dfcfdf50498b6742e2b3 not found: ID does not exist"
Jul 28 20:27:34.281221 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.281207    2578 scope.go:117] "RemoveContainer" containerID="94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c"
Jul 28 20:27:34.290199 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.290147    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"]
Jul 28 20:27:34.290199 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.290158    2578 scope.go:117] "RemoveContainer" containerID="94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c"
Jul 28 20:27:34.290607 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:27:34.290571    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c\": container with ID starting with 94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c not found: ID does not exist" containerID="94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c"
Jul 28 20:27:34.290697 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.290618    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c"} err="failed to get container status \"94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c\": rpc error: code = NotFound desc = could not find container \"94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c\": container with ID starting with 94cec870d3592b89c7a044d02be00dc29926c0a5f98cbdf02662320d0587ba3c not found: ID does not exist"
Jul 28 20:27:34.293457 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.293424    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f632f1-5fd4b6fd7d-p7t6d"]
Jul 28 20:27:34.308347 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.308318    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"]
Jul 28 20:27:34.312321 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.312296    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f632f1-68df47cdbf-hdsf7"]
Jul 28 20:27:34.323872 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.323846    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"]
Jul 28 20:27:34.329205 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.329173    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f632f1-99b598d4d-bmhg8"]
Jul 28 20:27:34.338664 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.338633    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1caddf2d-246e-41f0-a97a-48390d5da35d-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:34.338664 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.338662    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdfds\" (UniqueName: \"kubernetes.io/projected/1caddf2d-246e-41f0-a97a-48390d5da35d-kube-api-access-cdfds\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:34.338808 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:34.338680    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27vkw\" (UniqueName: \"kubernetes.io/projected/86100670-1faf-47bc-835b-2347e17eae45-kube-api-access-27vkw\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:27:35.651044 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:35.651006    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1caddf2d-246e-41f0-a97a-48390d5da35d" path="/var/lib/kubelet/pods/1caddf2d-246e-41f0-a97a-48390d5da35d/volumes"
Jul 28 20:27:35.651458 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:35.651388    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c84703-2327-4528-9f53-4639b5eae034" path="/var/lib/kubelet/pods/23c84703-2327-4528-9f53-4639b5eae034/volumes"
Jul 28 20:27:35.651707 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:35.651693    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86100670-1faf-47bc-835b-2347e17eae45" path="/var/lib/kubelet/pods/86100670-1faf-47bc-835b-2347e17eae45/volumes"
Jul 28 20:27:38.601430 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:38.601385    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:27:42.307007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.306965    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"]
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307291    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f780897e-9461-4465-9606-8a0aa54ce9d4" containerName="payload-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307304    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="f780897e-9461-4465-9606-8a0aa54ce9d4" containerName="payload-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307313    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1caddf2d-246e-41f0-a97a-48390d5da35d" containerName="payload-pre-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307318    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="1caddf2d-246e-41f0-a97a-48390d5da35d" containerName="payload-pre-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307331    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307336    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307342    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86100670-1faf-47bc-835b-2347e17eae45" containerName="payload-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307347    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="86100670-1faf-47bc-835b-2347e17eae45" containerName="payload-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307356    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c0dc692-c941-4852-ad55-6010fffcd8b7" containerName="payload-pre-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307361    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c0dc692-c941-4852-ad55-6010fffcd8b7" containerName="payload-pre-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307367    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c84703-2327-4528-9f53-4639b5eae034" containerName="maas-api"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307372    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c84703-2327-4528-9f53-4639b5eae034" containerName="maas-api"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307382    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39467efb-4f82-4245-86ee-51e15f66babe" containerName="maas-api"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307386    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="39467efb-4f82-4245-86ee-51e15f66babe" containerName="maas-api"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307438    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307445    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="f780897e-9461-4465-9606-8a0aa54ce9d4" containerName="payload-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307452    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307459    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="39467efb-4f82-4245-86ee-51e15f66babe" containerName="maas-api"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307465    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c84703-2327-4528-9f53-4639b5eae034" containerName="maas-api"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307470    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="1caddf2d-246e-41f0-a97a-48390d5da35d" containerName="payload-pre-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307476    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c0dc692-c941-4852-ad55-6010fffcd8b7" containerName="payload-pre-processing"
Jul 28 20:27:42.307570 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.307480    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="86100670-1faf-47bc-835b-2347e17eae45" containerName="payload-processing"
Jul 28 20:27:42.312105 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.312084    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.320289 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.320248    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ed54db-openshift-default-dockercfg-wzjsx\""
Jul 28 20:27:42.346377 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.346341    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"]
Jul 28 20:27:42.409787 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409732    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-token\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.409787 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afe3139-f01f-403f-9b07-0d3a39022dbb-istiod-ca-cert\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409805    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-podinfo\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409871    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-envoy\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409904    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdlhq\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-kube-api-access-zdlhq\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-socket\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.409977    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-certs\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.410005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-data\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.410045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.410029    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-credential-socket\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510746 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-certs\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510958 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-data\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510958 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510788    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-credential-socket\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510958 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510817    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-token\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510958 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510839    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afe3139-f01f-403f-9b07-0d3a39022dbb-istiod-ca-cert\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510958 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510861    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-podinfo\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.510958 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510884    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-envoy\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.510974    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdlhq\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-kube-api-access-zdlhq\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.511069    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-socket\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.511166    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-certs\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.511230    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-credential-socket\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.511258    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-data\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511442 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.511364    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-socket\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.511724 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.511659    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afe3139-f01f-403f-9b07-0d3a39022dbb-istiod-ca-cert\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.513493 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.513471    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-envoy\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.513652 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.513633    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-podinfo\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.529066 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.529029    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-token\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.529219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.529106    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdlhq\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-kube-api-access-zdlhq\") pod \"e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") " pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.622640 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.622520    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:42.787529 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.787502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"]
Jul 28 20:27:42.789660 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:27:42.789634    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8afe3139_f01f_403f_9b07_0d3a39022dbb.slice/crio-7673ec7047aa8aba963137f8779fe906a99a615c72726e8a14f635c5755f3b01 WatchSource:0}: Error finding container 7673ec7047aa8aba963137f8779fe906a99a615c72726e8a14f635c5755f3b01: Status 404 returned error can't find the container with id 7673ec7047aa8aba963137f8779fe906a99a615c72726e8a14f635c5755f3b01
Jul 28 20:27:42.791896 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.791865    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:27:42.791989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.791936    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:27:42.791989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:42.791966    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:27:43.311239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:43.311200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" event={"ID":"8afe3139-f01f-403f-9b07-0d3a39022dbb","Type":"ContainerStarted","Data":"f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"}
Jul 28 20:27:43.311691 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:43.311259    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" event={"ID":"8afe3139-f01f-403f-9b07-0d3a39022dbb","Type":"ContainerStarted","Data":"7673ec7047aa8aba963137f8779fe906a99a615c72726e8a14f635c5755f3b01"}
Jul 28 20:27:43.623355 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:43.623233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:43.628529 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:43.628500    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:43.669850 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:43.669793    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podStartSLOduration=1.669774992 podStartE2EDuration="1.669774992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:27:43.340619052 +0000 UTC m=+2466.240835894" watchObservedRunningTime="2026-07-28 20:27:43.669774992 +0000 UTC m=+2466.569991834"
Jul 28 20:27:44.316959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:44.316922    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:44.318087 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:44.318065    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:45.611985 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.611948    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"]
Jul 28 20:27:45.612403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.612305    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy"
Jul 28 20:27:45.612403 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.612319    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c52f5574-851d-4692-84a8-a0b9ab767b49" containerName="istio-proxy"
Jul 28 20:27:45.615157 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.615138    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.617717 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.617690    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5t8gx\""
Jul 28 20:27:45.618540 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.618519    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-ed54db-serving-cert\""
Jul 28 20:27:45.618824 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.618807    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 20:27:45.618892 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.618876    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 20:27:45.635822 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.635787    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"]
Jul 28 20:27:45.739637 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.739591    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1693d8d-6ec8-4a51-a2de-579563c69490-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.739835 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.739663    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1693d8d-6ec8-4a51-a2de-579563c69490-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.739835 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.739696    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpfm\" (UniqueName: \"kubernetes.io/projected/e1693d8d-6ec8-4a51-a2de-579563c69490-kube-api-access-hvpfm\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.840672 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.840626    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1693d8d-6ec8-4a51-a2de-579563c69490-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.840864 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.840681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpfm\" (UniqueName: \"kubernetes.io/projected/e1693d8d-6ec8-4a51-a2de-579563c69490-kube-api-access-hvpfm\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.840864 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.840729    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1693d8d-6ec8-4a51-a2de-579563c69490-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.841431 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.841407    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1693d8d-6ec8-4a51-a2de-579563c69490-openshift-service-ca\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.843326 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.843306    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1693d8d-6ec8-4a51-a2de-579563c69490-maas-api-tls\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.853482 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.853450    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpfm\" (UniqueName: \"kubernetes.io/projected/e1693d8d-6ec8-4a51-a2de-579563c69490-kube-api-access-hvpfm\") pod \"maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:45.926204 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:45.926101    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:46.089390 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.089340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"]
Jul 28 20:27:46.090738 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:27:46.090707    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1693d8d_6ec8_4a51_a2de_579563c69490.slice/crio-0590219cb1da8026a811b75b02698879a4751beef2e52b839584aa8168826acb WatchSource:0}: Error finding container 0590219cb1da8026a811b75b02698879a4751beef2e52b839584aa8168826acb: Status 404 returned error can't find the container with id 0590219cb1da8026a811b75b02698879a4751beef2e52b839584aa8168826acb
Jul 28 20:27:46.327726 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.327688    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9" event={"ID":"e1693d8d-6ec8-4a51-a2de-579563c69490","Type":"ContainerStarted","Data":"0590219cb1da8026a811b75b02698879a4751beef2e52b839584aa8168826acb"}
Jul 28 20:27:46.703407 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.703364    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"]
Jul 28 20:27:46.706745 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.706718    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:46.710034 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.710010    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-ed54db\""
Jul 28 20:27:46.736305 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.736251    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"]
Jul 28 20:27:46.785631 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.785592    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"]
Jul 28 20:27:46.789006 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.788987    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:46.796561 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.796536    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-ed54db-dockercfg-bvgnh\""
Jul 28 20:27:46.827165 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.827099    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"]
Jul 28 20:27:46.850949 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.850858    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:46.850949 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.850910    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqdvk\" (UniqueName: \"kubernetes.io/projected/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-kube-api-access-nqdvk\") pod \"payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:46.951941 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.951903    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4shd\" (UniqueName: \"kubernetes.io/projected/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-kube-api-access-n4shd\") pod \"payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:46.952099 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.951952    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:46.952099 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.951979    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:46.952099 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.952014    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqdvk\" (UniqueName: \"kubernetes.io/projected/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-kube-api-access-nqdvk\") pod \"payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:46.952709 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.952689    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:46.967629 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:46.967596    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqdvk\" (UniqueName: \"kubernetes.io/projected/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-kube-api-access-nqdvk\") pod \"payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:47.017966 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.017920    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:47.053636 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.053590    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4shd\" (UniqueName: \"kubernetes.io/projected/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-kube-api-access-n4shd\") pod \"payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:47.053846 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.053669    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:47.054458 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.054428    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:47.074864 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.074827    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4shd\" (UniqueName: \"kubernetes.io/projected/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-kube-api-access-n4shd\") pod \"payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") " pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:47.099431 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.099394    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:47.184949 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.184912    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"]
Jul 28 20:27:47.188256 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:27:47.188228    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2a73b7f_7ca5_406a_aeaa_7bcb26cdf39f.slice/crio-a580ea8d863c0465b2bf9b515be580f33e4fde336a0ddfd706259febc32e0e33 WatchSource:0}: Error finding container a580ea8d863c0465b2bf9b515be580f33e4fde336a0ddfd706259febc32e0e33: Status 404 returned error can't find the container with id a580ea8d863c0465b2bf9b515be580f33e4fde336a0ddfd706259febc32e0e33
Jul 28 20:27:47.270993 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.270959    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"]
Jul 28 20:27:47.273892 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:27:47.273864    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4304ed_f61f_4e90_9c80_d4d294f1e9bb.slice/crio-27fa79f9dcc56d75a63b3c58ced31cd7c65634260ec2e0dad73b679f3af6076b WatchSource:0}: Error finding container 27fa79f9dcc56d75a63b3c58ced31cd7c65634260ec2e0dad73b679f3af6076b: Status 404 returned error can't find the container with id 27fa79f9dcc56d75a63b3c58ced31cd7c65634260ec2e0dad73b679f3af6076b
Jul 28 20:27:47.335629 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.335592    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt" event={"ID":"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb","Type":"ContainerStarted","Data":"27fa79f9dcc56d75a63b3c58ced31cd7c65634260ec2e0dad73b679f3af6076b"}
Jul 28 20:27:47.337197 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.337168    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9" event={"ID":"e1693d8d-6ec8-4a51-a2de-579563c69490","Type":"ContainerStarted","Data":"6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7"}
Jul 28 20:27:47.337373 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.337230    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:47.338788 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.338760    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt" event={"ID":"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f","Type":"ContainerStarted","Data":"ff65e9bd19cc991fc75917c7bdb1e0491f84ecdcf0be6b276d25ae371b51b6ea"}
Jul 28 20:27:47.338885 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.338814    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:47.338885 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.338837    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt" event={"ID":"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f","Type":"ContainerStarted","Data":"a580ea8d863c0465b2bf9b515be580f33e4fde336a0ddfd706259febc32e0e33"}
Jul 28 20:27:47.369762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.369659    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9" podStartSLOduration=1.929471897 podStartE2EDuration="2.369642079s" totalImagesPullingTime="440.170182ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:27:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:27:47.366453067 +0000 UTC m=+2470.266669910" watchObservedRunningTime="2026-07-28 20:27:47.369642079 +0000 UTC m=+2470.269858920"
Jul 28 20:27:47.402596 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:47.402520    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt" podStartSLOduration=1.4024983469999999 podStartE2EDuration="1.402498347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:27:47.399775848 +0000 UTC m=+2470.299992703" watchObservedRunningTime="2026-07-28 20:27:47.402498347 +0000 UTC m=+2470.302715188"
Jul 28 20:27:48.346621 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:48.346579    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt" event={"ID":"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb","Type":"ContainerStarted","Data":"4c262b6f0f8c97a626b42524f02e11a393489b8c171d768c8ac5b5da506addce"}
Jul 28 20:27:48.386460 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:48.386400    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt" podStartSLOduration=2.3863797780000002 podStartE2EDuration="2.386379778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:27:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:27:48.385329764 +0000 UTC m=+2471.285546603" watchObservedRunningTime="2026-07-28 20:27:48.386379778 +0000 UTC m=+2471.286596620"
Jul 28 20:27:49.351632 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:49.351601    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:27:49.573087 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:49.573040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:27:53.352433 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:53.352402    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:27:58.349229 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.349200    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:27:58.403166 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.403131    2578 generic.go:356] "Generic (PLEG): container finished" podID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerID="f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11" exitCode=0
Jul 28 20:27:58.403410 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.403200    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" event={"ID":"8afe3139-f01f-403f-9b07-0d3a39022dbb","Type":"ContainerDied","Data":"f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"}
Jul 28 20:27:58.403683 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.403662    2578 scope.go:117] "RemoveContainer" containerID="f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"
Jul 28 20:27:58.404961 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.404927    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:27:58.405048 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.405006    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:27:58.405048 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:58.405040    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:27:59.317319 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:59.317233    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:59.410172 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:59.410133    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" event={"ID":"8afe3139-f01f-403f-9b07-0d3a39022dbb","Type":"ContainerStarted","Data":"399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b"}
Jul 28 20:27:59.624043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:59.623939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:27:59.625283 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:59.625221    2578 patch_prober.go:28] interesting pod/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 20:27:59.625458 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:27:59.625308    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 20:28:00.360477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:00.360437    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:28:00.623595 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:00.623492    2578 patch_prober.go:28] interesting pod/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 20:28:00.624178 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:00.624143    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 20:28:01.623893 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:01.623852    2578 patch_prober.go:28] interesting pod/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 20:28:01.624603 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:01.624567    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 20:28:02.623428 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:02.623378    2578 patch_prober.go:28] interesting pod/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 20:28:02.623723 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:02.623693    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 20:28:03.627891 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:03.627856    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:28:03.628421 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:03.628313    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:28:03.629013 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:03.628994    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:28:14.787801 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.787758    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"]
Jul 28 20:28:14.790829 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.790803    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.793681 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.793657    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-1231ba-serving-cert\""
Jul 28 20:28:14.824426 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.824389    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"]
Jul 28 20:28:14.884733 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.884691    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfwhb\" (UniqueName: \"kubernetes.io/projected/45915c69-8608-4cb5-a5f0-876d086c5431-kube-api-access-gfwhb\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.884919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.884777    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45915c69-8608-4cb5-a5f0-876d086c5431-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.884919 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.884802    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.986237 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.986194    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfwhb\" (UniqueName: \"kubernetes.io/projected/45915c69-8608-4cb5-a5f0-876d086c5431-kube-api-access-gfwhb\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.986463 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.986309    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45915c69-8608-4cb5-a5f0-876d086c5431-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.986463 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.986341    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:14.986463 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:28:14.986457    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-serving-cert: secret "maas-api-e2e-shared-b-1231ba-serving-cert" not found
Jul 28 20:28:14.986587 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:28:14.986534    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls podName:45915c69-8608-4cb5-a5f0-876d086c5431 nodeName:}" failed. No retries permitted until 2026-07-28 20:28:15.486518859 +0000 UTC m=+2498.386735679 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls") pod "maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" (UID: "45915c69-8608-4cb5-a5f0-876d086c5431") : secret "maas-api-e2e-shared-b-1231ba-serving-cert" not found
Jul 28 20:28:14.987044 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:14.987020    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45915c69-8608-4cb5-a5f0-876d086c5431-openshift-service-ca\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:15.003342 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:15.003302    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfwhb\" (UniqueName: \"kubernetes.io/projected/45915c69-8608-4cb5-a5f0-876d086c5431-kube-api-access-gfwhb\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:15.490314 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:15.490262    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:15.492897 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:15.492876    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls\") pod \"maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:15.702050 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:15.701998    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:15.890445 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:15.890410    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"]
Jul 28 20:28:15.894002 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:28:15.893971    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45915c69_8608_4cb5_a5f0_876d086c5431.slice/crio-57da1d256867bf828d0b46334cd253457ac4dc5e7f4512cb96c913af72b6569f WatchSource:0}: Error finding container 57da1d256867bf828d0b46334cd253457ac4dc5e7f4512cb96c913af72b6569f: Status 404 returned error can't find the container with id 57da1d256867bf828d0b46334cd253457ac4dc5e7f4512cb96c913af72b6569f
Jul 28 20:28:16.251069 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.251046    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"]
Jul 28 20:28:16.253491 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.253475    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.257371 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.257351    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-1231ba-dockercfg-zjvv5\""
Jul 28 20:28:16.257613 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.257592    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-1231ba\""
Jul 28 20:28:16.267242 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.267214    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"]
Jul 28 20:28:16.270357 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.270338    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.293333 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.293293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"]
Jul 28 20:28:16.295966 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.295934    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"]
Jul 28 20:28:16.298190 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.298164    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chzz5\" (UniqueName: \"kubernetes.io/projected/713fb5ff-30b0-4186-97ad-cb00b0376f0a-kube-api-access-chzz5\") pod \"payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.298357 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.298214    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rhq\" (UniqueName: \"kubernetes.io/projected/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-kube-api-access-g5rhq\") pod \"payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.298357 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.298328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.298357 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.298347    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713fb5ff-30b0-4186-97ad-cb00b0376f0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.399163 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.399124    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chzz5\" (UniqueName: \"kubernetes.io/projected/713fb5ff-30b0-4186-97ad-cb00b0376f0a-kube-api-access-chzz5\") pod \"payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.399378 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.399173    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rhq\" (UniqueName: \"kubernetes.io/projected/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-kube-api-access-g5rhq\") pod \"payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.399378 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.399226    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.399378 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.399251    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713fb5ff-30b0-4186-97ad-cb00b0376f0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.399957 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.399932    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.400074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.400047    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713fb5ff-30b0-4186-97ad-cb00b0376f0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.414106 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.414077    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chzz5\" (UniqueName: \"kubernetes.io/projected/713fb5ff-30b0-4186-97ad-cb00b0376f0a-kube-api-access-chzz5\") pod \"payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.415888 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.415861    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rhq\" (UniqueName: \"kubernetes.io/projected/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-kube-api-access-g5rhq\") pod \"payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.514140 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.514104    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" event={"ID":"45915c69-8608-4cb5-a5f0-876d086c5431","Type":"ContainerStarted","Data":"49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3"}
Jul 28 20:28:16.514379 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.514166    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:16.514379 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.514183    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" event={"ID":"45915c69-8608-4cb5-a5f0-876d086c5431","Type":"ContainerStarted","Data":"57da1d256867bf828d0b46334cd253457ac4dc5e7f4512cb96c913af72b6569f"}
Jul 28 20:28:16.590741 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.590678    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" podStartSLOduration=2.238694858 podStartE2EDuration="2.590655162s" totalImagesPullingTime="351.960304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:16.582494315 +0000 UTC m=+2499.482711160" watchObservedRunningTime="2026-07-28 20:28:16.590655162 +0000 UTC m=+2499.490872004"
Jul 28 20:28:16.594687 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.594664    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:16.614526 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.614482    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:16.776477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.776445    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"]
Jul 28 20:28:16.778130 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:28:16.778105    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614e919e_06ce_4636_b40a_0cfa3f0cc0c1.slice/crio-15d47ce2ba03bc64fd57b35ff33bab0e17e5d5092f22abd799321741d6719c50 WatchSource:0}: Error finding container 15d47ce2ba03bc64fd57b35ff33bab0e17e5d5092f22abd799321741d6719c50: Status 404 returned error can't find the container with id 15d47ce2ba03bc64fd57b35ff33bab0e17e5d5092f22abd799321741d6719c50
Jul 28 20:28:16.797930 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:16.797903    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"]
Jul 28 20:28:16.812520 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:28:16.812489    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713fb5ff_30b0_4186_97ad_cb00b0376f0a.slice/crio-a3bdc00fcf6d3d0b629001e9663c0111ef2476830a3edeeff616a9489bbc6761 WatchSource:0}: Error finding container a3bdc00fcf6d3d0b629001e9663c0111ef2476830a3edeeff616a9489bbc6761: Status 404 returned error can't find the container with id a3bdc00fcf6d3d0b629001e9663c0111ef2476830a3edeeff616a9489bbc6761
Jul 28 20:28:17.522635 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.522598    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6" event={"ID":"713fb5ff-30b0-4186-97ad-cb00b0376f0a","Type":"ContainerStarted","Data":"5008d124fefa50087d5e8893278e6b77ca3e7d0a4025d79ddc6f21a9157e7fce"}
Jul 28 20:28:17.523142 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.522664    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6" event={"ID":"713fb5ff-30b0-4186-97ad-cb00b0376f0a","Type":"ContainerStarted","Data":"a3bdc00fcf6d3d0b629001e9663c0111ef2476830a3edeeff616a9489bbc6761"}
Jul 28 20:28:17.523142 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.522783    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:17.524586 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.524547    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8" event={"ID":"614e919e-06ce-4636-b40a-0cfa3f0cc0c1","Type":"ContainerStarted","Data":"61841b8a735931d41a28b1c78488a9d3017e723fe8968192c6752cfa3485d7bd"}
Jul 28 20:28:17.524735 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.524615    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8" event={"ID":"614e919e-06ce-4636-b40a-0cfa3f0cc0c1","Type":"ContainerStarted","Data":"15d47ce2ba03bc64fd57b35ff33bab0e17e5d5092f22abd799321741d6719c50"}
Jul 28 20:28:17.559045 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.558987    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6" podStartSLOduration=1.558968415 podStartE2EDuration="1.558968415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:17.557921979 +0000 UTC m=+2500.458138821" watchObservedRunningTime="2026-07-28 20:28:17.558968415 +0000 UTC m=+2500.459185256"
Jul 28 20:28:17.603565 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:17.603515    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8" podStartSLOduration=1.60349924 podStartE2EDuration="1.60349924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:28:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:28:17.602672982 +0000 UTC m=+2500.502889823" watchObservedRunningTime="2026-07-28 20:28:17.60349924 +0000 UTC m=+2500.503716082"
Jul 28 20:28:18.530967 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:18.530927    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:28:21.675329 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:21.675278    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:28:22.530652 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:22.530620    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:28:28.533193 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:28.533158    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:28:29.538312 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:28:29.538219    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:29:01.568384 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:29:01.568340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:29:22.783752 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:29:22.783709    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:29:56.777457 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:29:56.777420    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:30:17.409304 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:30:17.409250    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:30:35.393579 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:30:35.393537    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:30:56.400595 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:30:56.400563    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:31:13.982458 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:31:13.982415    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:31:34.181532 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:31:34.181437    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:31:37.773803 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:31:37.773770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:31:37.779811 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:31:37.779784    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:31:52.007321 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:31:52.007279    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:32:12.280616 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:32:12.280576    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:32:30.175567 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:32:30.175523    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:32:51.387112 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:32:51.387070    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:33:08.579532 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:33:08.579438    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:33:36.493820 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:33:36.493778    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:33:52.883337 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:33:52.883293    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:34:10.099323 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:34:10.099256    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:34:33.567784 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:34:33.567695    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:35:09.869626 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:35:09.869588    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:35:29.674731 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:35:29.674693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:35:46.786574 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:35:46.786528    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:35:56.680309 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:35:56.680255    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:36:37.808593 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:36:37.808563    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:36:37.815638 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:36:37.815610    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:36:38.306050 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:36:38.306013    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:37:36.086779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:37:36.086693    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:37:50.877682 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:37:50.877635    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:38:05.507477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:05.507435    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:38:21.777131 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:21.777090    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:38:29.542007 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.541967    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"]
Jul 28 20:38:29.545620 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.545600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.548374 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.548344    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-92ba10-openshift-default-dockercfg-gvjkb\""
Jul 28 20:38:29.558724 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.558684    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"]
Jul 28 20:38:29.617920 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.617880    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-podinfo\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.617920 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.617924    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-socket\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.617948    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-certs\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.618047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-data\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.618102    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-envoy\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.618122    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262167d6-b39d-4b18-af23-68af6cbe3f43-istiod-ca-cert\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618303 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.618144    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjlmh\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-kube-api-access-rjlmh\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618303 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.618169    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-token\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.618303 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.618218    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-credential-socket\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.718914 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.718871    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-envoy\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.718914 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.718911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262167d6-b39d-4b18-af23-68af6cbe3f43-istiod-ca-cert\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.718932    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjlmh\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-kube-api-access-rjlmh\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.718967    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-token\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.718995    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-credential-socket\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719048    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-podinfo\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-socket\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719104    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-certs\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719187 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719159    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-data\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719610 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719525    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-credential-socket\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719610 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719589    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-socket\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719750    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-data\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719821 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719759    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-certs\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.719868 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.719852    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262167d6-b39d-4b18-af23-68af6cbe3f43-istiod-ca-cert\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.722135 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.722102    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-envoy\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.722250 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.722217    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-podinfo\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.728126 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.728094    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-token\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.728244 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.728219    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjlmh\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-kube-api-access-rjlmh\") pod \"e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") " pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:29.857054 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:29.856966    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:30.008409 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.008040    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"]
Jul 28 20:38:30.011447 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:38:30.011411    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262167d6_b39d_4b18_af23_68af6cbe3f43.slice/crio-e7ae4d0b2c10acdc11c60195471cf497d318c54202138469b3227f7169422972 WatchSource:0}: Error finding container e7ae4d0b2c10acdc11c60195471cf497d318c54202138469b3227f7169422972: Status 404 returned error can't find the container with id e7ae4d0b2c10acdc11c60195471cf497d318c54202138469b3227f7169422972
Jul 28 20:38:30.014156 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.014118    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:38:30.014240 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.014208    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:38:30.014305 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.014249    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:38:30.838649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.838606    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" event={"ID":"262167d6-b39d-4b18-af23-68af6cbe3f43","Type":"ContainerStarted","Data":"cdf509e6c173d70199d871b2dbcc15657a4c96b25ffaf135070d25795b70bc98"}
Jul 28 20:38:30.839054 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.838675    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" event={"ID":"262167d6-b39d-4b18-af23-68af6cbe3f43","Type":"ContainerStarted","Data":"e7ae4d0b2c10acdc11c60195471cf497d318c54202138469b3227f7169422972"}
Jul 28 20:38:30.857436 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.857400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:30.864858 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:30.864804    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" podStartSLOduration=1.8647850190000002 podStartE2EDuration="1.864785019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:30.862579735 +0000 UTC m=+3113.762796602" watchObservedRunningTime="2026-07-28 20:38:30.864785019 +0000 UTC m=+3113.765001861"
Jul 28 20:38:31.863111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:31.863077    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:32.849213 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:32.849176    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:32.850110 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:32.850087    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:33.940441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:33.940406    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"]
Jul 28 20:38:33.943907 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:33.943887    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:33.947296 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:33.947253    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-92ba10\""
Jul 28 20:38:33.947441 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:33.947423    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-92ba10-dockercfg-xwwdc\""
Jul 28 20:38:33.958806 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:33.958774    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"]
Jul 28 20:38:34.063768 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.063721    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9r62\" (UniqueName: \"kubernetes.io/projected/9771615a-66ae-4029-ac44-9251e4ba1793-kube-api-access-h9r62\") pod \"payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.063950 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.063789    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9771615a-66ae-4029-ac44-9251e4ba1793-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.164907 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.164860    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9r62\" (UniqueName: \"kubernetes.io/projected/9771615a-66ae-4029-ac44-9251e4ba1793-kube-api-access-h9r62\") pod \"payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.165135 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.164948    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9771615a-66ae-4029-ac44-9251e4ba1793-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.165724 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.165702    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9771615a-66ae-4029-ac44-9251e4ba1793-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.173960 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.173921    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9r62\" (UniqueName: \"kubernetes.io/projected/9771615a-66ae-4029-ac44-9251e4ba1793-kube-api-access-h9r62\") pod \"payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.255223 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.255126    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.409110 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.409083    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"]
Jul 28 20:38:34.411928 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:38:34.411897    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9771615a_66ae_4029_ac44_9251e4ba1793.slice/crio-b5020b8d54dbebc18479beecdd126932368bddf7c3e83fe7cabd1fbc9cf62b0a WatchSource:0}: Error finding container b5020b8d54dbebc18479beecdd126932368bddf7c3e83fe7cabd1fbc9cf62b0a: Status 404 returned error can't find the container with id b5020b8d54dbebc18479beecdd126932368bddf7c3e83fe7cabd1fbc9cf62b0a
Jul 28 20:38:34.862452 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.862410    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj" event={"ID":"9771615a-66ae-4029-ac44-9251e4ba1793","Type":"ContainerStarted","Data":"a18aabc9099d273e3cac8c498da83250f2b1344f350d4177d49d4179e01862bb"}
Jul 28 20:38:34.862663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.862596    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:34.862663 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.862631    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj" event={"ID":"9771615a-66ae-4029-ac44-9251e4ba1793","Type":"ContainerStarted","Data":"b5020b8d54dbebc18479beecdd126932368bddf7c3e83fe7cabd1fbc9cf62b0a"}
Jul 28 20:38:34.883122 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:34.883059    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj" podStartSLOduration=1.883031972 podStartE2EDuration="1.883031972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:38:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:38:34.881691411 +0000 UTC m=+3117.781908253" watchObservedRunningTime="2026-07-28 20:38:34.883031972 +0000 UTC m=+3117.783248814"
Jul 28 20:38:37.478633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:37.478591    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:38:44.919806 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:44.919772    2578 generic.go:356] "Generic (PLEG): container finished" podID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerID="cdf509e6c173d70199d871b2dbcc15657a4c96b25ffaf135070d25795b70bc98" exitCode=0
Jul 28 20:38:44.920245 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:44.919851    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" event={"ID":"262167d6-b39d-4b18-af23-68af6cbe3f43","Type":"ContainerDied","Data":"cdf509e6c173d70199d871b2dbcc15657a4c96b25ffaf135070d25795b70bc98"}
Jul 28 20:38:44.920311 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:44.920257    2578 scope.go:117] "RemoveContainer" containerID="cdf509e6c173d70199d871b2dbcc15657a4c96b25ffaf135070d25795b70bc98"
Jul 28 20:38:44.921314 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:44.921258    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:38:44.921422 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:44.921357    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:38:44.921422 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:44.921391    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:38:45.871321 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:45.871285    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:38:45.936704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:45.936656    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" event={"ID":"262167d6-b39d-4b18-af23-68af6cbe3f43","Type":"ContainerStarted","Data":"8689b1a98feb878301c34747f3e991cf75da647eb57bb6c034c5cd28d7ff888c"}
Jul 28 20:38:46.857689 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:46.857644    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:46.858849 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:46.858823    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Jul 28 20:38:46.858912 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:46.858887    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Jul 28 20:38:47.849979 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:47.849939    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:47.858358 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:47.858328    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Jul 28 20:38:47.858547 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:47.858390    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Jul 28 20:38:48.858104 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:48.858065    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Jul 28 20:38:48.858542 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:48.858126    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Jul 28 20:38:49.857991 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:49.857947    2578 patch_prober.go:28] interesting pod/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Jul 28 20:38:49.858252 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:49.858016    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Jul 28 20:38:50.861908 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:50.861874    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:38:50.862905 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:38:50.862887    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:39:02.081453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.081362    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"]
Jul 28 20:39:02.085079 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.085057    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.087965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.087944    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-37887f-serving-cert\""
Jul 28 20:39:02.097893 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.097862    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"]
Jul 28 20:39:02.135777 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.135731    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.135961 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.135801    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.135961 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.135874    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmpr\" (UniqueName: \"kubernetes.io/projected/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-kube-api-access-pvmpr\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.237013 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.236964    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.237204 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.237042    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.237204 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.237086    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvmpr\" (UniqueName: \"kubernetes.io/projected/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-kube-api-access-pvmpr\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.237204 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:39:02.237132    2578 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-serving-cert: secret "maas-api-e2e-ipp-b-37887f-serving-cert" not found
Jul 28 20:39:02.237204 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:39:02.237202    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls podName:7ca21b13-8bee-47c0-be50-ccb1702d3fb5 nodeName:}" failed. No retries permitted until 2026-07-28 20:39:02.737180604 +0000 UTC m=+3145.637397442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls") pod "maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" (UID: "7ca21b13-8bee-47c0-be50-ccb1702d3fb5") : secret "maas-api-e2e-ipp-b-37887f-serving-cert" not found
Jul 28 20:39:02.237805 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.237786    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.250085 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.250049    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvmpr\" (UniqueName: \"kubernetes.io/projected/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-kube-api-access-pvmpr\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.743348 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.743254    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.746060 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.746026    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls\") pod \"maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:02.790951 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.790911    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"]
Jul 28 20:39:02.795984 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.794896    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:02.801639 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.798865    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-37887f-dockercfg-cp5pz\""
Jul 28 20:39:02.801639 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.799291    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-37887f\""
Jul 28 20:39:02.805993 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.805954    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"]
Jul 28 20:39:02.844690 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.844647    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"]
Jul 28 20:39:02.848148 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.848125    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:02.859383 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.859340    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"]
Jul 28 20:39:02.945093 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.945047    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md8h9\" (UniqueName: \"kubernetes.io/projected/c4697002-f9b3-49c7-bf9b-ee7d10866031-kube-api-access-md8h9\") pod \"payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:02.945327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.945172    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6w49\" (UniqueName: \"kubernetes.io/projected/2bf465df-f192-461f-a3e7-d45363b6ebea-kube-api-access-s6w49\") pod \"payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:02.945327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.945228    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4697002-f9b3-49c7-bf9b-ee7d10866031-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:02.945416 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.945333    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf465df-f192-461f-a3e7-d45363b6ebea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:02.996298 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:02.996176    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:03.046713 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.046671    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md8h9\" (UniqueName: \"kubernetes.io/projected/c4697002-f9b3-49c7-bf9b-ee7d10866031-kube-api-access-md8h9\") pod \"payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:03.046934 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.046783    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6w49\" (UniqueName: \"kubernetes.io/projected/2bf465df-f192-461f-a3e7-d45363b6ebea-kube-api-access-s6w49\") pod \"payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:03.046934 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.046816    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4697002-f9b3-49c7-bf9b-ee7d10866031-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:03.046934 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.046869    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf465df-f192-461f-a3e7-d45363b6ebea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:03.047649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.047622    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf465df-f192-461f-a3e7-d45363b6ebea-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:03.047760 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.047679    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4697002-f9b3-49c7-bf9b-ee7d10866031-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:03.056537 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.056484    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6w49\" (UniqueName: \"kubernetes.io/projected/2bf465df-f192-461f-a3e7-d45363b6ebea-kube-api-access-s6w49\") pod \"payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:03.057197 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.057164    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md8h9\" (UniqueName: \"kubernetes.io/projected/c4697002-f9b3-49c7-bf9b-ee7d10866031-kube-api-access-md8h9\") pod \"payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:03.111816 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.111777    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:03.152908 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.152877    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"]
Jul 28 20:39:03.155707 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:39:03.155657    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca21b13_8bee_47c0_be50_ccb1702d3fb5.slice/crio-313215f3679f011b5457b953d98a6aec8c40bc43dee4636a788fd37396de6efa WatchSource:0}: Error finding container 313215f3679f011b5457b953d98a6aec8c40bc43dee4636a788fd37396de6efa: Status 404 returned error can't find the container with id 313215f3679f011b5457b953d98a6aec8c40bc43dee4636a788fd37396de6efa
Jul 28 20:39:03.157175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.157155    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 20:39:03.160002 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.159893    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:03.274538 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.274494    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"]
Jul 28 20:39:03.277599 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:39:03.277539    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf465df_f192_461f_a3e7_d45363b6ebea.slice/crio-83cd0be463c1096dc59b9970637f8216a778a152a605a6eea22bc2ce240d5f96 WatchSource:0}: Error finding container 83cd0be463c1096dc59b9970637f8216a778a152a605a6eea22bc2ce240d5f96: Status 404 returned error can't find the container with id 83cd0be463c1096dc59b9970637f8216a778a152a605a6eea22bc2ce240d5f96
Jul 28 20:39:03.334161 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:03.334133    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"]
Jul 28 20:39:03.337503 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:39:03.337475    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4697002_f9b3_49c7_bf9b_ee7d10866031.slice/crio-ad4469f4534b0955ac3229053322ccd394a5da4672cbcb9963c81597adbd1f76 WatchSource:0}: Error finding container ad4469f4534b0955ac3229053322ccd394a5da4672cbcb9963c81597adbd1f76: Status 404 returned error can't find the container with id ad4469f4534b0955ac3229053322ccd394a5da4672cbcb9963c81597adbd1f76
Jul 28 20:39:04.040706 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.040668    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq" event={"ID":"2bf465df-f192-461f-a3e7-d45363b6ebea","Type":"ContainerStarted","Data":"5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb"}
Jul 28 20:39:04.040901 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.040731    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq" event={"ID":"2bf465df-f192-461f-a3e7-d45363b6ebea","Type":"ContainerStarted","Data":"83cd0be463c1096dc59b9970637f8216a778a152a605a6eea22bc2ce240d5f96"}
Jul 28 20:39:04.040901 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.040868    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:04.042414 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.042386    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" event={"ID":"7ca21b13-8bee-47c0-be50-ccb1702d3fb5","Type":"ContainerStarted","Data":"ba8f732946d1bcce2bf3c0fbfbc02a69a7f18f094dc94517bbc2ae4de226ba53"}
Jul 28 20:39:04.042594 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.042430    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" event={"ID":"7ca21b13-8bee-47c0-be50-ccb1702d3fb5","Type":"ContainerStarted","Data":"313215f3679f011b5457b953d98a6aec8c40bc43dee4636a788fd37396de6efa"}
Jul 28 20:39:04.042594 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.042460    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:04.044039 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.044013    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4" event={"ID":"c4697002-f9b3-49c7-bf9b-ee7d10866031","Type":"ContainerStarted","Data":"c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb"}
Jul 28 20:39:04.044142 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.044060    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4" event={"ID":"c4697002-f9b3-49c7-bf9b-ee7d10866031","Type":"ContainerStarted","Data":"ad4469f4534b0955ac3229053322ccd394a5da4672cbcb9963c81597adbd1f76"}
Jul 28 20:39:04.044142 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.044120    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:04.060552 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.060488    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq" podStartSLOduration=2.060463563 podStartE2EDuration="2.060463563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:39:04.058247872 +0000 UTC m=+3146.958464716" watchObservedRunningTime="2026-07-28 20:39:04.060463563 +0000 UTC m=+3146.960680408"
Jul 28 20:39:04.112432 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.112366    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" podStartSLOduration=1.749679375 podStartE2EDuration="2.112344481s" totalImagesPullingTime="362.665106ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:39:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:39:04.084743921 +0000 UTC m=+3146.984960764" watchObservedRunningTime="2026-07-28 20:39:04.112344481 +0000 UTC m=+3147.012561326"
Jul 28 20:39:04.113037 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:04.113002    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4" podStartSLOduration=2.112993667 podStartE2EDuration="2.112993667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:39:04.1094095 +0000 UTC m=+3147.009626342" watchObservedRunningTime="2026-07-28 20:39:04.112993667 +0000 UTC m=+3147.013210510"
Jul 28 20:39:10.057332 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:10.057293    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:39:11.467372 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:11.467332    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:39:15.053446 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:15.053400    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:39:15.053881 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:15.053842    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:39:34.203247 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:34.203211    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq_2bf465df-f192-461f-a3e7-d45363b6ebea/payload-processing/0.log"
Jul 28 20:39:46.875518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:39:46.875470    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:40:13.679733 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:13.679680    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:40:26.827358 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:26.827324    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq_2bf465df-f192-461f-a3e7-d45363b6ebea/payload-processing/0.log"
Jul 28 20:40:27.529786 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.527514    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"]
Jul 28 20:40:27.534452 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.534420    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.538114 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.538092    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-579f3e-openshift-default-dockercfg-tgf2v\""
Jul 28 20:40:27.556512 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.556469    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"]
Jul 28 20:40:27.624921 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.624885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-credential-socket\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.624931    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-socket\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.624961    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdcv4\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-kube-api-access-bdcv4\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.625051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-istio-token\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625130 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.625114    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2830-e487-434a-8041-1758fdf2f255-istiod-ca-cert\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625395 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.625155    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2830-e487-434a-8041-1758fdf2f255-istio-podinfo\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625395 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.625202    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-data\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625395 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.625235    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-certs\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.625395 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.625320    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-envoy\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726196 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726156    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2830-e487-434a-8041-1758fdf2f255-istiod-ca-cert\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726196 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726202    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2830-e487-434a-8041-1758fdf2f255-istio-podinfo\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726476 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-data\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726476 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726397    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-certs\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726476 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726442    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-envoy\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726642 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726543    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-credential-socket\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726642 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726600    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-socket\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdcv4\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-kube-api-access-bdcv4\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726738 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726681    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-istio-token\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726842    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-data\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.726938 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.726873    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-certs\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.727051 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.727021    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2830-e487-434a-8041-1758fdf2f255-istiod-ca-cert\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.727291 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.727241    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-socket\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.727540 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.727515    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-credential-socket\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.728942 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.728919    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-envoy\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.729024 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.728938    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2830-e487-434a-8041-1758fdf2f255-istio-podinfo\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.742715 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.742680    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdcv4\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-kube-api-access-bdcv4\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.744180 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.744155    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-istio-token\") pod \"e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") " pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:27.845869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:27.845762    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:28.000565 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.000539    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"]
Jul 28 20:40:28.002134 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:40:28.002094    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ac2830_e487_434a_8041_1758fdf2f255.slice/crio-7b59dfeeebae3690d1c9e25f61dff9d537a28a522471db9ae44c502ffbdb8a47 WatchSource:0}: Error finding container 7b59dfeeebae3690d1c9e25f61dff9d537a28a522471db9ae44c502ffbdb8a47: Status 404 returned error can't find the container with id 7b59dfeeebae3690d1c9e25f61dff9d537a28a522471db9ae44c502ffbdb8a47
Jul 28 20:40:28.004351 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.004316    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:40:28.004453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.004390    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:40:28.004453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.004421    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:40:28.517302 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.517245    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" event={"ID":"b9ac2830-e487-434a-8041-1758fdf2f255","Type":"ContainerStarted","Data":"8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"}
Jul 28 20:40:28.517495 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.517326    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" event={"ID":"b9ac2830-e487-434a-8041-1758fdf2f255","Type":"ContainerStarted","Data":"7b59dfeeebae3690d1c9e25f61dff9d537a28a522471db9ae44c502ffbdb8a47"}
Jul 28 20:40:28.588751 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.588688    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podStartSLOduration=1.588668922 podStartE2EDuration="1.588668922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 20:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:40:28.586377795 +0000 UTC m=+3231.486594636" watchObservedRunningTime="2026-07-28 20:40:28.588668922 +0000 UTC m=+3231.488885763"
Jul 28 20:40:28.846963 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:28.846861    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:29.852006 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:29.851973    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:30.529254 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:30.529222    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:30.530355 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:30.530332    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:42.095485 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:42.095444    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:40:50.648973 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:50.648933    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9ac2830-e487-434a-8041-1758fdf2f255" containerID="8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa" exitCode=0
Jul 28 20:40:50.649502 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:50.649005    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" event={"ID":"b9ac2830-e487-434a-8041-1758fdf2f255","Type":"ContainerDied","Data":"8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"}
Jul 28 20:40:50.649502 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:50.649425    2578 scope.go:117] "RemoveContainer" containerID="8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"
Jul 28 20:40:50.650520 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:50.650490    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:40:50.650635 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:50.650577    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:40:50.650635 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:50.650611    2578 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 28 20:40:51.655894 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:51.655861    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" event={"ID":"b9ac2830-e487-434a-8041-1758fdf2f255","Type":"ContainerStarted","Data":"5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7"}
Jul 28 20:40:51.845986 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:51.845944    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:51.846208 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:51.846187    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Jul 28 20:40:51.846292 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:51.846237    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Jul 28 20:40:52.847156 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:52.847112    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Jul 28 20:40:52.847620 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:52.847187    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Jul 28 20:40:53.846432 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:53.846387    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Jul 28 20:40:53.846633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:53.846447    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Jul 28 20:40:54.846903 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:54.846854    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Jul 28 20:40:54.847340 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:54.846920    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Jul 28 20:40:55.846395 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:55.846349    2578 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Jul 28 20:40:55.846634 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:55.846421    2578 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Jul 28 20:40:56.850926 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:56.850893    2578 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:56.851425 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:56.851316    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:40:56.852081 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:40:56.852061    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:41:16.992252 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:16.992209    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:41:19.737417 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:19.737256    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"]
Jul 28 20:41:19.737970 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:19.737538    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy" containerID="cri-o://5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7" gracePeriod=30
Jul 28 20:41:25.000867 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.000829    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:41:25.082190 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082154    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdcv4\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-kube-api-access-bdcv4\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082190 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082189    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2830-e487-434a-8041-1758fdf2f255-istio-podinfo\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082219    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-socket\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082238    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2830-e487-434a-8041-1758fdf2f255-istiod-ca-cert\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082309    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-certs\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082327    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-envoy\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082560    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-istio-token\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082574    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:25.082704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082645    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-credential-socket\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082683    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-data\") pod \"b9ac2830-e487-434a-8041-1758fdf2f255\" (UID: \"b9ac2830-e487-434a-8041-1758fdf2f255\") "
Jul 28 20:41:25.082704 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082680    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:25.083219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082815    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ac2830-e487-434a-8041-1758fdf2f255-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:41:25.083219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.082911    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:25.083219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.083078    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.083219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.083099    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9ac2830-e487-434a-8041-1758fdf2f255-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.083219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.083115    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.083219 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.083127    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.084069 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.084040    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-data" (OuterVolumeSpecName: "istio-data") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:25.084805 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.084775    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:41:25.084805 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.084787    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9ac2830-e487-434a-8041-1758fdf2f255-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:41:25.085239 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.085212    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-kube-api-access-bdcv4" (OuterVolumeSpecName: "kube-api-access-bdcv4") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "kube-api-access-bdcv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:25.085361 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.085239    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-istio-token" (OuterVolumeSpecName: "istio-token") pod "b9ac2830-e487-434a-8041-1758fdf2f255" (UID: "b9ac2830-e487-434a-8041-1758fdf2f255"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:25.183834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.183776    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.183834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.183827    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdcv4\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-kube-api-access-bdcv4\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.183834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.183836    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9ac2830-e487-434a-8041-1758fdf2f255-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.183834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.183845    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9ac2830-e487-434a-8041-1758fdf2f255-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.183834 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.183854    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9ac2830-e487-434a-8041-1758fdf2f255-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:25.846257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.846223    2578 generic.go:356] "Generic (PLEG): container finished" podID="b9ac2830-e487-434a-8041-1758fdf2f255" containerID="5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7" exitCode=0
Jul 28 20:41:25.846467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.846299    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" event={"ID":"b9ac2830-e487-434a-8041-1758fdf2f255","Type":"ContainerDied","Data":"5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7"}
Jul 28 20:41:25.846467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.846303    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"
Jul 28 20:41:25.846467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.846356    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr" event={"ID":"b9ac2830-e487-434a-8041-1758fdf2f255","Type":"ContainerDied","Data":"7b59dfeeebae3690d1c9e25f61dff9d537a28a522471db9ae44c502ffbdb8a47"}
Jul 28 20:41:25.846467 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.846379    2578 scope.go:117] "RemoveContainer" containerID="5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7"
Jul 28 20:41:25.856250 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.856228    2578 scope.go:117] "RemoveContainer" containerID="8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"
Jul 28 20:41:25.866132 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.866092    2578 scope.go:117] "RemoveContainer" containerID="5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7"
Jul 28 20:41:25.866497 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:41:25.866477    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7\": container with ID starting with 5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7 not found: ID does not exist" containerID="5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7"
Jul 28 20:41:25.866598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.866507    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7"} err="failed to get container status \"5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7\": rpc error: code = NotFound desc = could not find container \"5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7\": container with ID starting with 5eff79e4c5cc064f617f799853be816f23e388ed84030b27dde0b32da87599a7 not found: ID does not exist"
Jul 28 20:41:25.866598 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.866540    2578 scope.go:117] "RemoveContainer" containerID="8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"
Jul 28 20:41:25.866874 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:41:25.866846    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa\": container with ID starting with 8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa not found: ID does not exist" containerID="8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"
Jul 28 20:41:25.866938 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.866884    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa"} err="failed to get container status \"8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa\": rpc error: code = NotFound desc = could not find container \"8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa\": container with ID starting with 8794314039c97f0f4c5d8d9fa132cbc6e13df9d20133519d48aea9a0b6655bfa not found: ID does not exist"
Jul 28 20:41:25.868125 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.868101    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"]
Jul 28 20:41:25.873385 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:25.873358    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-579f3e-openshift-default-76c58d7746-4lzpr"]
Jul 28 20:41:27.658965 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:27.658928    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" path="/var/lib/kubelet/pods/b9ac2830-e487-434a-8041-1758fdf2f255/volumes"
Jul 28 20:41:37.846494 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:37.846462    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:41:37.854843 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:37.854816    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:41:38.311539 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:38.311502    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:41:39.654051 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.654016    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"]
Jul 28 20:41:39.654558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.654236    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj" podUID="9771615a-66ae-4029-ac44-9251e4ba1793" containerName="payload-pre-processing" containerID="cri-o://a18aabc9099d273e3cac8c498da83250f2b1344f350d4177d49d4179e01862bb" gracePeriod=30
Jul 28 20:41:39.928812 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.928775    2578 generic.go:356] "Generic (PLEG): container finished" podID="9771615a-66ae-4029-ac44-9251e4ba1793" containerID="a18aabc9099d273e3cac8c498da83250f2b1344f350d4177d49d4179e01862bb" exitCode=0
Jul 28 20:41:39.928955 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.928848    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj" event={"ID":"9771615a-66ae-4029-ac44-9251e4ba1793","Type":"ContainerDied","Data":"a18aabc9099d273e3cac8c498da83250f2b1344f350d4177d49d4179e01862bb"}
Jul 28 20:41:39.928955 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.928904    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj" event={"ID":"9771615a-66ae-4029-ac44-9251e4ba1793","Type":"ContainerDied","Data":"b5020b8d54dbebc18479beecdd126932368bddf7c3e83fe7cabd1fbc9cf62b0a"}
Jul 28 20:41:39.928955 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.928922    2578 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5020b8d54dbebc18479beecdd126932368bddf7c3e83fe7cabd1fbc9cf62b0a"
Jul 28 20:41:39.937840 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:39.937815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:41:40.022569 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.022499    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9771615a-66ae-4029-ac44-9251e4ba1793-plugins-config-volume\") pod \"9771615a-66ae-4029-ac44-9251e4ba1793\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") "
Jul 28 20:41:40.022797 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.022603    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9r62\" (UniqueName: \"kubernetes.io/projected/9771615a-66ae-4029-ac44-9251e4ba1793-kube-api-access-h9r62\") pod \"9771615a-66ae-4029-ac44-9251e4ba1793\" (UID: \"9771615a-66ae-4029-ac44-9251e4ba1793\") "
Jul 28 20:41:40.023003 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.022978    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9771615a-66ae-4029-ac44-9251e4ba1793-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9771615a-66ae-4029-ac44-9251e4ba1793" (UID: "9771615a-66ae-4029-ac44-9251e4ba1793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:41:40.024922 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.024891    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9771615a-66ae-4029-ac44-9251e4ba1793-kube-api-access-h9r62" (OuterVolumeSpecName: "kube-api-access-h9r62") pod "9771615a-66ae-4029-ac44-9251e4ba1793" (UID: "9771615a-66ae-4029-ac44-9251e4ba1793"). InnerVolumeSpecName "kube-api-access-h9r62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:41:40.123486 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.123449    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9r62\" (UniqueName: \"kubernetes.io/projected/9771615a-66ae-4029-ac44-9251e4ba1793-kube-api-access-h9r62\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:40.123486 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.123480    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9771615a-66ae-4029-ac44-9251e4ba1793-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:41:40.934453 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.934416    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"
Jul 28 20:41:40.972739 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.972700    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"]
Jul 28 20:41:40.980157 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:40.980117    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-92ba10-545bd876fb-5sfdj"]
Jul 28 20:41:41.653148 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:41.653105    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9771615a-66ae-4029-ac44-9251e4ba1793" path="/var/lib/kubelet/pods/9771615a-66ae-4029-ac44-9251e4ba1793/volumes"
Jul 28 20:41:49.202685 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:41:49.202629    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:42:40.096063 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:40.096022    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"]
Jul 28 20:42:40.096656 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:40.096240    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy" containerID="cri-o://8689b1a98feb878301c34747f3e991cf75da647eb57bb6c034c5cd28d7ff888c" gracePeriod=30
Jul 28 20:42:45.303338 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.303259    2578 generic.go:356] "Generic (PLEG): container finished" podID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerID="8689b1a98feb878301c34747f3e991cf75da647eb57bb6c034c5cd28d7ff888c" exitCode=0
Jul 28 20:42:45.303338 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.303317    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" event={"ID":"262167d6-b39d-4b18-af23-68af6cbe3f43","Type":"ContainerDied","Data":"8689b1a98feb878301c34747f3e991cf75da647eb57bb6c034c5cd28d7ff888c"}
Jul 28 20:42:45.303929 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.303446    2578 scope.go:117] "RemoveContainer" containerID="cdf509e6c173d70199d871b2dbcc15657a4c96b25ffaf135070d25795b70bc98"
Jul 28 20:42:45.386013 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.385985    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:42:45.453536 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453500    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-credential-socket\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453553    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-socket\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453572    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjlmh\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-kube-api-access-rjlmh\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453608    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262167d6-b39d-4b18-af23-68af6cbe3f43-istiod-ca-cert\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453635    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-data\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453664    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-certs\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453707    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-token\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.453734 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453736    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-podinfo\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.454137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453765    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-envoy\") pod \"262167d6-b39d-4b18-af23-68af6cbe3f43\" (UID: \"262167d6-b39d-4b18-af23-68af6cbe3f43\") "
Jul 28 20:42:45.454137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453796    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:45.454137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.453851    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:45.454137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.454001    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:45.454137 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.454120    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.454446 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.454141    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.454446 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.454155    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.454843 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.454785    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262167d6-b39d-4b18-af23-68af6cbe3f43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:45.455085 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.455055    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-data" (OuterVolumeSpecName: "istio-data") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:45.456828 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.456772    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-token" (OuterVolumeSpecName: "istio-token") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:45.456921 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.456885    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:42:45.456970 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.456933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:42:45.456970 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.456937    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-kube-api-access-rjlmh" (OuterVolumeSpecName: "kube-api-access-rjlmh") pod "262167d6-b39d-4b18-af23-68af6cbe3f43" (UID: "262167d6-b39d-4b18-af23-68af6cbe3f43"). InnerVolumeSpecName "kube-api-access-rjlmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:45.555327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.555257    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.555327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.555323    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.555327 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.555332    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.555587 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.555341    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjlmh\" (UniqueName: \"kubernetes.io/projected/262167d6-b39d-4b18-af23-68af6cbe3f43-kube-api-access-rjlmh\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.555587 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.555351    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/262167d6-b39d-4b18-af23-68af6cbe3f43-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:45.555587 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:45.555358    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/262167d6-b39d-4b18-af23-68af6cbe3f43-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.106575 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.106534    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"]
Jul 28 20:42:46.106798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.106757    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" podUID="7ca21b13-8bee-47c0-be50-ccb1702d3fb5" containerName="maas-api" containerID="cri-o://ba8f732946d1bcce2bf3c0fbfbc02a69a7f18f094dc94517bbc2ae4de226ba53" gracePeriod=30
Jul 28 20:42:46.269719 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.269680    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"]
Jul 28 20:42:46.269955 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.269922    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq" podUID="2bf465df-f192-461f-a3e7-d45363b6ebea" containerName="payload-processing" containerID="cri-o://5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb" gracePeriod=30
Jul 28 20:42:46.287811 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.287776    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"]
Jul 28 20:42:46.288083 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.288013    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4" podUID="c4697002-f9b3-49c7-bf9b-ee7d10866031" containerName="payload-pre-processing" containerID="cri-o://c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb" gracePeriod=30
Jul 28 20:42:46.317897 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.317860    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk" event={"ID":"262167d6-b39d-4b18-af23-68af6cbe3f43","Type":"ContainerDied","Data":"e7ae4d0b2c10acdc11c60195471cf497d318c54202138469b3227f7169422972"}
Jul 28 20:42:46.318370 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.317923    2578 scope.go:117] "RemoveContainer" containerID="8689b1a98feb878301c34747f3e991cf75da647eb57bb6c034c5cd28d7ff888c"
Jul 28 20:42:46.318370 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.318029    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"
Jul 28 20:42:46.320609 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.320581    2578 generic.go:356] "Generic (PLEG): container finished" podID="7ca21b13-8bee-47c0-be50-ccb1702d3fb5" containerID="ba8f732946d1bcce2bf3c0fbfbc02a69a7f18f094dc94517bbc2ae4de226ba53" exitCode=0
Jul 28 20:42:46.320751 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.320672    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" event={"ID":"7ca21b13-8bee-47c0-be50-ccb1702d3fb5","Type":"ContainerDied","Data":"ba8f732946d1bcce2bf3c0fbfbc02a69a7f18f094dc94517bbc2ae4de226ba53"}
Jul 28 20:42:46.419071 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.419040    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:42:46.445093 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.445057    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"]
Jul 28 20:42:46.456244 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.456212    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-92ba10-openshift-default-566b564dbd-c9shk"]
Jul 28 20:42:46.564297 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.564247    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls\") pod \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") "
Jul 28 20:42:46.564481 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.564347    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-openshift-service-ca\") pod \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") "
Jul 28 20:42:46.564556 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.564486    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvmpr\" (UniqueName: \"kubernetes.io/projected/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-kube-api-access-pvmpr\") pod \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\" (UID: \"7ca21b13-8bee-47c0-be50-ccb1702d3fb5\") "
Jul 28 20:42:46.564701 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.564670    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ca21b13-8bee-47c0-be50-ccb1702d3fb5" (UID: "7ca21b13-8bee-47c0-be50-ccb1702d3fb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:46.565212 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.565185    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.566882 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.566855    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ca21b13-8bee-47c0-be50-ccb1702d3fb5" (UID: "7ca21b13-8bee-47c0-be50-ccb1702d3fb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:42:46.566991 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.566968    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-kube-api-access-pvmpr" (OuterVolumeSpecName: "kube-api-access-pvmpr") pod "7ca21b13-8bee-47c0-be50-ccb1702d3fb5" (UID: "7ca21b13-8bee-47c0-be50-ccb1702d3fb5"). InnerVolumeSpecName "kube-api-access-pvmpr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:46.579002 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.578974    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:42:46.580952 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.580929    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:42:46.666303 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666169    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6w49\" (UniqueName: \"kubernetes.io/projected/2bf465df-f192-461f-a3e7-d45363b6ebea-kube-api-access-s6w49\") pod \"2bf465df-f192-461f-a3e7-d45363b6ebea\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") "
Jul 28 20:42:46.666303 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666233    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf465df-f192-461f-a3e7-d45363b6ebea-plugins-config-volume\") pod \"2bf465df-f192-461f-a3e7-d45363b6ebea\" (UID: \"2bf465df-f192-461f-a3e7-d45363b6ebea\") "
Jul 28 20:42:46.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666307    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md8h9\" (UniqueName: \"kubernetes.io/projected/c4697002-f9b3-49c7-bf9b-ee7d10866031-kube-api-access-md8h9\") pod \"c4697002-f9b3-49c7-bf9b-ee7d10866031\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") "
Jul 28 20:42:46.666563 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666331    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4697002-f9b3-49c7-bf9b-ee7d10866031-plugins-config-volume\") pod \"c4697002-f9b3-49c7-bf9b-ee7d10866031\" (UID: \"c4697002-f9b3-49c7-bf9b-ee7d10866031\") "
Jul 28 20:42:46.666682 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666618    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.666682 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666637    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvmpr\" (UniqueName: \"kubernetes.io/projected/7ca21b13-8bee-47c0-be50-ccb1702d3fb5-kube-api-access-pvmpr\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.666798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666743    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bf465df-f192-461f-a3e7-d45363b6ebea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2bf465df-f192-461f-a3e7-d45363b6ebea" (UID: "2bf465df-f192-461f-a3e7-d45363b6ebea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:46.666798 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.666770    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4697002-f9b3-49c7-bf9b-ee7d10866031-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c4697002-f9b3-49c7-bf9b-ee7d10866031" (UID: "c4697002-f9b3-49c7-bf9b-ee7d10866031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:42:46.668709 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.668684    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4697002-f9b3-49c7-bf9b-ee7d10866031-kube-api-access-md8h9" (OuterVolumeSpecName: "kube-api-access-md8h9") pod "c4697002-f9b3-49c7-bf9b-ee7d10866031" (UID: "c4697002-f9b3-49c7-bf9b-ee7d10866031"). InnerVolumeSpecName "kube-api-access-md8h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:46.668773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.668691    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bf465df-f192-461f-a3e7-d45363b6ebea-kube-api-access-s6w49" (OuterVolumeSpecName: "kube-api-access-s6w49") pod "2bf465df-f192-461f-a3e7-d45363b6ebea" (UID: "2bf465df-f192-461f-a3e7-d45363b6ebea"). InnerVolumeSpecName "kube-api-access-s6w49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:42:46.767444 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.767405    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6w49\" (UniqueName: \"kubernetes.io/projected/2bf465df-f192-461f-a3e7-d45363b6ebea-kube-api-access-s6w49\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.767444 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.767444    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2bf465df-f192-461f-a3e7-d45363b6ebea-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.767652 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.767460    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md8h9\" (UniqueName: \"kubernetes.io/projected/c4697002-f9b3-49c7-bf9b-ee7d10866031-kube-api-access-md8h9\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:46.767652 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:46.767476    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4697002-f9b3-49c7-bf9b-ee7d10866031-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:42:47.327133 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.327093    2578 generic.go:356] "Generic (PLEG): container finished" podID="c4697002-f9b3-49c7-bf9b-ee7d10866031" containerID="c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb" exitCode=0
Jul 28 20:42:47.327633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.327140    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"
Jul 28 20:42:47.327633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.327185    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4" event={"ID":"c4697002-f9b3-49c7-bf9b-ee7d10866031","Type":"ContainerDied","Data":"c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb"}
Jul 28 20:42:47.327633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.327242    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4" event={"ID":"c4697002-f9b3-49c7-bf9b-ee7d10866031","Type":"ContainerDied","Data":"ad4469f4534b0955ac3229053322ccd394a5da4672cbcb9963c81597adbd1f76"}
Jul 28 20:42:47.327633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.327288    2578 scope.go:117] "RemoveContainer" containerID="c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb"
Jul 28 20:42:47.329840 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.329805    2578 generic.go:356] "Generic (PLEG): container finished" podID="2bf465df-f192-461f-a3e7-d45363b6ebea" containerID="5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb" exitCode=0
Jul 28 20:42:47.329989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.329845    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"
Jul 28 20:42:47.329989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.329899    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq" event={"ID":"2bf465df-f192-461f-a3e7-d45363b6ebea","Type":"ContainerDied","Data":"5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb"}
Jul 28 20:42:47.329989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.329956    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq" event={"ID":"2bf465df-f192-461f-a3e7-d45363b6ebea","Type":"ContainerDied","Data":"83cd0be463c1096dc59b9970637f8216a778a152a605a6eea22bc2ce240d5f96"}
Jul 28 20:42:47.331402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.331374    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m" event={"ID":"7ca21b13-8bee-47c0-be50-ccb1702d3fb5","Type":"ContainerDied","Data":"313215f3679f011b5457b953d98a6aec8c40bc43dee4636a788fd37396de6efa"}
Jul 28 20:42:47.331402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.331402    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"
Jul 28 20:42:47.339183 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.339158    2578 scope.go:117] "RemoveContainer" containerID="c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb"
Jul 28 20:42:47.339931 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:42:47.339899    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb\": container with ID starting with c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb not found: ID does not exist" containerID="c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb"
Jul 28 20:42:47.340043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.339943    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb"} err="failed to get container status \"c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb\": rpc error: code = NotFound desc = could not find container \"c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb\": container with ID starting with c7e72a4af51687f3b6fa8c09591798a682e3843a9d596abc42297ca171fe8cbb not found: ID does not exist"
Jul 28 20:42:47.340043 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.339970    2578 scope.go:117] "RemoveContainer" containerID="5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb"
Jul 28 20:42:47.349854 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.349833    2578 scope.go:117] "RemoveContainer" containerID="5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb"
Jul 28 20:42:47.350146 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:42:47.350127    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb\": container with ID starting with 5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb not found: ID does not exist" containerID="5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb"
Jul 28 20:42:47.350191 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.350157    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb"} err="failed to get container status \"5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb\": rpc error: code = NotFound desc = could not find container \"5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb\": container with ID starting with 5570a5d3ca40621704cfc1d504ca0fab5ade066b7fd3cca94ddcb3863c9b4ecb not found: ID does not exist"
Jul 28 20:42:47.350191 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.350175    2578 scope.go:117] "RemoveContainer" containerID="ba8f732946d1bcce2bf3c0fbfbc02a69a7f18f094dc94517bbc2ae4de226ba53"
Jul 28 20:42:47.388364 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.388260    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"]
Jul 28 20:42:47.398487 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.398453    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-37887f-787db5b4b4-4xbl4"]
Jul 28 20:42:47.428792 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.428748    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"]
Jul 28 20:42:47.434062 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.434014    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-37887f-5d89b9b667-4hn2m"]
Jul 28 20:42:47.460103 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.460060    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"]
Jul 28 20:42:47.469211 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.469181    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-37887f-756bd49798-hmrfq"]
Jul 28 20:42:47.652216 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.652130    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" path="/var/lib/kubelet/pods/262167d6-b39d-4b18-af23-68af6cbe3f43/volumes"
Jul 28 20:42:47.652549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.652535    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bf465df-f192-461f-a3e7-d45363b6ebea" path="/var/lib/kubelet/pods/2bf465df-f192-461f-a3e7-d45363b6ebea/volumes"
Jul 28 20:42:47.652876 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.652864    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ca21b13-8bee-47c0-be50-ccb1702d3fb5" path="/var/lib/kubelet/pods/7ca21b13-8bee-47c0-be50-ccb1702d3fb5/volumes"
Jul 28 20:42:47.653530 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:47.653515    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4697002-f9b3-49c7-bf9b-ee7d10866031" path="/var/lib/kubelet/pods/c4697002-f9b3-49c7-bf9b-ee7d10866031/volumes"
Jul 28 20:42:49.721429 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:42:49.721387    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:43:06.059989 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.059946    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"]
Jul 28 20:43:06.060463 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.060153    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9" podUID="e1693d8d-6ec8-4a51-a2de-579563c69490" containerName="maas-api" containerID="cri-o://6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7" gracePeriod=30
Jul 28 20:43:06.285281 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.285229    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"]
Jul 28 20:43:06.285490 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.285462    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt" podUID="fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" containerName="payload-processing" containerID="cri-o://4c262b6f0f8c97a626b42524f02e11a393489b8c171d768c8ac5b5da506addce" gracePeriod=30
Jul 28 20:43:06.298472 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.298441    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"]
Jul 28 20:43:06.298712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.298688    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt" podUID="e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" containerName="payload-pre-processing" containerID="cri-o://ff65e9bd19cc991fc75917c7bdb1e0491f84ecdcf0be6b276d25ae371b51b6ea" gracePeriod=30
Jul 28 20:43:06.423335 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.423304    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:43:06.441405 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.441177    2578 generic.go:356] "Generic (PLEG): container finished" podID="fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" containerID="4c262b6f0f8c97a626b42524f02e11a393489b8c171d768c8ac5b5da506addce" exitCode=0
Jul 28 20:43:06.441405 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.441306    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt" event={"ID":"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb","Type":"ContainerDied","Data":"4c262b6f0f8c97a626b42524f02e11a393489b8c171d768c8ac5b5da506addce"}
Jul 28 20:43:06.441762 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.441735    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1693d8d-6ec8-4a51-a2de-579563c69490-maas-api-tls\") pod \"e1693d8d-6ec8-4a51-a2de-579563c69490\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") "
Jul 28 20:43:06.441915 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.441896    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1693d8d-6ec8-4a51-a2de-579563c69490-openshift-service-ca\") pod \"e1693d8d-6ec8-4a51-a2de-579563c69490\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") "
Jul 28 20:43:06.441995 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.441932    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvpfm\" (UniqueName: \"kubernetes.io/projected/e1693d8d-6ec8-4a51-a2de-579563c69490-kube-api-access-hvpfm\") pod \"e1693d8d-6ec8-4a51-a2de-579563c69490\" (UID: \"e1693d8d-6ec8-4a51-a2de-579563c69490\") "
Jul 28 20:43:06.442980 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.442933    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1693d8d-6ec8-4a51-a2de-579563c69490-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1693d8d-6ec8-4a51-a2de-579563c69490" (UID: "e1693d8d-6ec8-4a51-a2de-579563c69490"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:06.443712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.443572    2578 generic.go:356] "Generic (PLEG): container finished" podID="e1693d8d-6ec8-4a51-a2de-579563c69490" containerID="6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7" exitCode=0
Jul 28 20:43:06.443712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.443634    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9" event={"ID":"e1693d8d-6ec8-4a51-a2de-579563c69490","Type":"ContainerDied","Data":"6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7"}
Jul 28 20:43:06.443712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.443704    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9" event={"ID":"e1693d8d-6ec8-4a51-a2de-579563c69490","Type":"ContainerDied","Data":"0590219cb1da8026a811b75b02698879a4751beef2e52b839584aa8168826acb"}
Jul 28 20:43:06.443942 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.443740    2578 scope.go:117] "RemoveContainer" containerID="6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7"
Jul 28 20:43:06.444005 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.443964    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"
Jul 28 20:43:06.445627 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.444947    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1693d8d-6ec8-4a51-a2de-579563c69490-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1693d8d-6ec8-4a51-a2de-579563c69490" (UID: "e1693d8d-6ec8-4a51-a2de-579563c69490"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:43:06.445627 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.445420    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1693d8d-6ec8-4a51-a2de-579563c69490-kube-api-access-hvpfm" (OuterVolumeSpecName: "kube-api-access-hvpfm") pod "e1693d8d-6ec8-4a51-a2de-579563c69490" (UID: "e1693d8d-6ec8-4a51-a2de-579563c69490"). InnerVolumeSpecName "kube-api-access-hvpfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:06.446812 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.446572    2578 generic.go:356] "Generic (PLEG): container finished" podID="e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" containerID="ff65e9bd19cc991fc75917c7bdb1e0491f84ecdcf0be6b276d25ae371b51b6ea" exitCode=0
Jul 28 20:43:06.446812 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.446653    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt" event={"ID":"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f","Type":"ContainerDied","Data":"ff65e9bd19cc991fc75917c7bdb1e0491f84ecdcf0be6b276d25ae371b51b6ea"}
Jul 28 20:43:06.462645 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.462531    2578 scope.go:117] "RemoveContainer" containerID="6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7"
Jul 28 20:43:06.462975 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:43:06.462909    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7\": container with ID starting with 6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7 not found: ID does not exist" containerID="6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7"
Jul 28 20:43:06.462975 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.462948    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7"} err="failed to get container status \"6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7\": rpc error: code = NotFound desc = could not find container \"6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7\": container with ID starting with 6800eda2566aab7d7607d32080c66052777ce122a6336be8cb2f6df74b25e2b7 not found: ID does not exist"
Jul 28 20:43:06.543004 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.542967    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1693d8d-6ec8-4a51-a2de-579563c69490-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.543004 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.543005    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1693d8d-6ec8-4a51-a2de-579563c69490-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.543248 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.543021    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvpfm\" (UniqueName: \"kubernetes.io/projected/e1693d8d-6ec8-4a51-a2de-579563c69490-kube-api-access-hvpfm\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.578676 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.578645    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:43:06.587051 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.587023    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:43:06.643744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.643641    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-plugins-config-volume\") pod \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") "
Jul 28 20:43:06.643744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.643685    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-plugins-config-volume\") pod \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") "
Jul 28 20:43:06.643744 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.643740    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4shd\" (UniqueName: \"kubernetes.io/projected/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-kube-api-access-n4shd\") pod \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\" (UID: \"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb\") "
Jul 28 20:43:06.643991 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.643834    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqdvk\" (UniqueName: \"kubernetes.io/projected/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-kube-api-access-nqdvk\") pod \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\" (UID: \"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f\") "
Jul 28 20:43:06.644168 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.644115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" (UID: "fc4304ed-f61f-4e90-9c80-d4d294f1e9bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:06.644168 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.644115    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" (UID: "e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:06.646330 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.646307    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-kube-api-access-nqdvk" (OuterVolumeSpecName: "kube-api-access-nqdvk") pod "e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" (UID: "e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f"). InnerVolumeSpecName "kube-api-access-nqdvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:06.646330 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.646319    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-kube-api-access-n4shd" (OuterVolumeSpecName: "kube-api-access-n4shd") pod "fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" (UID: "fc4304ed-f61f-4e90-9c80-d4d294f1e9bb"). InnerVolumeSpecName "kube-api-access-n4shd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:06.745195 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.745160    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.745195 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.745197    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.745195 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.745212    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4shd\" (UniqueName: \"kubernetes.io/projected/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb-kube-api-access-n4shd\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.745503 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.745227    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqdvk\" (UniqueName: \"kubernetes.io/projected/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f-kube-api-access-nqdvk\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:06.789257 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.789209    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"]
Jul 28 20:43:06.798681 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:06.798647    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-ed54db-6879946b69-jdkh9"]
Jul 28 20:43:07.453246 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.453147    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt" event={"ID":"fc4304ed-f61f-4e90-9c80-d4d294f1e9bb","Type":"ContainerDied","Data":"27fa79f9dcc56d75a63b3c58ced31cd7c65634260ec2e0dad73b679f3af6076b"}
Jul 28 20:43:07.453246 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.453238    2578 scope.go:117] "RemoveContainer" containerID="4c262b6f0f8c97a626b42524f02e11a393489b8c171d768c8ac5b5da506addce"
Jul 28 20:43:07.453246 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.453171    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"
Jul 28 20:43:07.455694 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.455671    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"
Jul 28 20:43:07.455806 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.455703    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt" event={"ID":"e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f","Type":"ContainerDied","Data":"a580ea8d863c0465b2bf9b515be580f33e4fde336a0ddfd706259febc32e0e33"}
Jul 28 20:43:07.463089 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.463071    2578 scope.go:117] "RemoveContainer" containerID="ff65e9bd19cc991fc75917c7bdb1e0491f84ecdcf0be6b276d25ae371b51b6ea"
Jul 28 20:43:07.487047 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.487008    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"]
Jul 28 20:43:07.503371 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.503333    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-ed54db-7847d48bf8-7gfrt"]
Jul 28 20:43:07.525338 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.525300    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"]
Jul 28 20:43:07.528759 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.528725    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-ed54db-6c55999b6d-vgppt"]
Jul 28 20:43:07.653484 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.653444    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1693d8d-6ec8-4a51-a2de-579563c69490" path="/var/lib/kubelet/pods/e1693d8d-6ec8-4a51-a2de-579563c69490/volumes"
Jul 28 20:43:07.653825 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.653808    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" path="/var/lib/kubelet/pods/e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f/volumes"
Jul 28 20:43:07.654151 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:07.654137    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" path="/var/lib/kubelet/pods/fc4304ed-f61f-4e90-9c80-d4d294f1e9bb/volumes"
Jul 28 20:43:08.200606 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:08.200565    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:43:14.163008 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:14.162966    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"]
Jul 28 20:43:14.163595 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:14.163195    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy" containerID="cri-o://399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b" gracePeriod=30
Jul 28 20:43:18.630797 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:18.630751    2578 patch_prober.go:28] interesting pod/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 20:43:18.631189 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:18.630827    2578 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 20:43:19.429145 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.429069    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:43:19.454922 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.454877    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-socket\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455042 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.454942    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-certs\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455042 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.454962    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-envoy\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455042 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.454986    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afe3139-f01f-403f-9b07-0d3a39022dbb-istiod-ca-cert\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455042 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455007    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-data\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455042 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455033    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdlhq\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-kube-api-access-zdlhq\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455309 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455075    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-token\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455309 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455106    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-podinfo\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455309 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455166    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:19.455309 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455175    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-credential-socket\") pod \"8afe3139-f01f-403f-9b07-0d3a39022dbb\" (UID: \"8afe3139-f01f-403f-9b07-0d3a39022dbb\") "
Jul 28 20:43:19.455519 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455438    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:19.455519 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455475    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:19.455638 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455555    2578 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.455638 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455577    2578 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-workload-certs\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.455638 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455592    2578 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-credential-socket\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.455803 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.455737    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8afe3139-f01f-403f-9b07-0d3a39022dbb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:19.456375 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.456337    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:19.457425 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.457397    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 20:43:19.457592 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.457567    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-kube-api-access-zdlhq" (OuterVolumeSpecName: "kube-api-access-zdlhq") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "kube-api-access-zdlhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:19.457671 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.457647    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 20:43:19.457780 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.457764    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8afe3139-f01f-403f-9b07-0d3a39022dbb" (UID: "8afe3139-f01f-403f-9b07-0d3a39022dbb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:19.525341 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.525307    2578 generic.go:356] "Generic (PLEG): container finished" podID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerID="399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b" exitCode=0
Jul 28 20:43:19.525518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.525375    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"
Jul 28 20:43:19.525518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.525397    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" event={"ID":"8afe3139-f01f-403f-9b07-0d3a39022dbb","Type":"ContainerDied","Data":"399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b"}
Jul 28 20:43:19.525518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.525481    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj" event={"ID":"8afe3139-f01f-403f-9b07-0d3a39022dbb","Type":"ContainerDied","Data":"7673ec7047aa8aba963137f8779fe906a99a615c72726e8a14f635c5755f3b01"}
Jul 28 20:43:19.525518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.525506    2578 scope.go:117] "RemoveContainer" containerID="399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b"
Jul 28 20:43:19.535750 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.535730    2578 scope.go:117] "RemoveContainer" containerID="f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"
Jul 28 20:43:19.545111 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.545091    2578 scope.go:117] "RemoveContainer" containerID="399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b"
Jul 28 20:43:19.545425 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:43:19.545403    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b\": container with ID starting with 399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b not found: ID does not exist" containerID="399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b"
Jul 28 20:43:19.545487 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.545435    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b"} err="failed to get container status \"399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b\": rpc error: code = NotFound desc = could not find container \"399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b\": container with ID starting with 399ddc3eaf760a037b4f2061c9360ec1acce17e28e5f78ccc2920394c639612b not found: ID does not exist"
Jul 28 20:43:19.545487 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.545454    2578 scope.go:117] "RemoveContainer" containerID="f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"
Jul 28 20:43:19.545737 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:43:19.545720    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11\": container with ID starting with f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11 not found: ID does not exist" containerID="f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"
Jul 28 20:43:19.545775 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.545744    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11"} err="failed to get container status \"f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11\": rpc error: code = NotFound desc = could not find container \"f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11\": container with ID starting with f8b42a834fa94148baae02fd34fb7701f6777e4ed0c832cae31c6510bfc71a11 not found: ID does not exist"
Jul 28 20:43:19.556074 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556041    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"]
Jul 28 20:43:19.556503 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556481    2578 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-envoy\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.556549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556508    2578 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8afe3139-f01f-403f-9b07-0d3a39022dbb-istiod-ca-cert\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.556549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556522    2578 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-data\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.556549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556530    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdlhq\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-kube-api-access-zdlhq\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.556549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556540    2578 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-token\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.556549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.556548    2578 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8afe3139-f01f-403f-9b07-0d3a39022dbb-istio-podinfo\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:19.567208 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.567179    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ed54db-openshift-default-769d867f5b-l2vhj"]
Jul 28 20:43:19.650906 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:19.650868    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" path="/var/lib/kubelet/pods/8afe3139-f01f-403f-9b07-0d3a39022dbb/volumes"
Jul 28 20:43:25.218066 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.218020    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"]
Jul 28 20:43:25.218636 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.218314    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" podUID="45915c69-8608-4cb5-a5f0-876d086c5431" containerName="maas-api" containerID="cri-o://49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3" gracePeriod=30
Jul 28 20:43:25.325216 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.325178    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"]
Jul 28 20:43:25.325487 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.325453    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8" podUID="614e919e-06ce-4636-b40a-0cfa3f0cc0c1" containerName="payload-processing" containerID="cri-o://61841b8a735931d41a28b1c78488a9d3017e723fe8968192c6752cfa3485d7bd" gracePeriod=30
Jul 28 20:43:25.340986 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.340952    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"]
Jul 28 20:43:25.341170 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.341148    2578 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6" podUID="713fb5ff-30b0-4186-97ad-cb00b0376f0a" containerName="payload-pre-processing" containerID="cri-o://5008d124fefa50087d5e8893278e6b77ca3e7d0a4025d79ddc6f21a9157e7fce" gracePeriod=30
Jul 28 20:43:25.490591 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.490562    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:43:25.562787 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.562745    2578 generic.go:356] "Generic (PLEG): container finished" podID="45915c69-8608-4cb5-a5f0-876d086c5431" containerID="49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3" exitCode=0
Jul 28 20:43:25.562981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.562815    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"
Jul 28 20:43:25.562981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.562840    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" event={"ID":"45915c69-8608-4cb5-a5f0-876d086c5431","Type":"ContainerDied","Data":"49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3"}
Jul 28 20:43:25.562981 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.562961    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9" event={"ID":"45915c69-8608-4cb5-a5f0-876d086c5431","Type":"ContainerDied","Data":"57da1d256867bf828d0b46334cd253457ac4dc5e7f4512cb96c913af72b6569f"}
Jul 28 20:43:25.563177 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.562994    2578 scope.go:117] "RemoveContainer" containerID="49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3"
Jul 28 20:43:25.565521 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.565181    2578 generic.go:356] "Generic (PLEG): container finished" podID="713fb5ff-30b0-4186-97ad-cb00b0376f0a" containerID="5008d124fefa50087d5e8893278e6b77ca3e7d0a4025d79ddc6f21a9157e7fce" exitCode=0
Jul 28 20:43:25.565521 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.565457    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6" event={"ID":"713fb5ff-30b0-4186-97ad-cb00b0376f0a","Type":"ContainerDied","Data":"5008d124fefa50087d5e8893278e6b77ca3e7d0a4025d79ddc6f21a9157e7fce"}
Jul 28 20:43:25.568027 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.567650    2578 generic.go:356] "Generic (PLEG): container finished" podID="614e919e-06ce-4636-b40a-0cfa3f0cc0c1" containerID="61841b8a735931d41a28b1c78488a9d3017e723fe8968192c6752cfa3485d7bd" exitCode=0
Jul 28 20:43:25.568027 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.567694    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8" event={"ID":"614e919e-06ce-4636-b40a-0cfa3f0cc0c1","Type":"ContainerDied","Data":"61841b8a735931d41a28b1c78488a9d3017e723fe8968192c6752cfa3485d7bd"}
Jul 28 20:43:25.576185 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.576156    2578 scope.go:117] "RemoveContainer" containerID="49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3"
Jul 28 20:43:25.576577 ip-10-0-133-160 kubenswrapper[2578]: E0728 20:43:25.576545    2578 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3\": container with ID starting with 49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3 not found: ID does not exist" containerID="49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3"
Jul 28 20:43:25.576692 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.576582    2578 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3"} err="failed to get container status \"49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3\": rpc error: code = NotFound desc = could not find container \"49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3\": container with ID starting with 49c20c0c59cd9083c111e11c003bf934a14031a6e2a6a72b54822585d8930af3 not found: ID does not exist"
Jul 28 20:43:25.608959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.608842    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfwhb\" (UniqueName: \"kubernetes.io/projected/45915c69-8608-4cb5-a5f0-876d086c5431-kube-api-access-gfwhb\") pod \"45915c69-8608-4cb5-a5f0-876d086c5431\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") "
Jul 28 20:43:25.608959 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.608925    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45915c69-8608-4cb5-a5f0-876d086c5431-openshift-service-ca\") pod \"45915c69-8608-4cb5-a5f0-876d086c5431\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") "
Jul 28 20:43:25.609370 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.609333    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls\") pod \"45915c69-8608-4cb5-a5f0-876d086c5431\" (UID: \"45915c69-8608-4cb5-a5f0-876d086c5431\") "
Jul 28 20:43:25.612655 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.612623    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45915c69-8608-4cb5-a5f0-876d086c5431-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "45915c69-8608-4cb5-a5f0-876d086c5431" (UID: "45915c69-8608-4cb5-a5f0-876d086c5431"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:25.621289 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.616376    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "45915c69-8608-4cb5-a5f0-876d086c5431" (UID: "45915c69-8608-4cb5-a5f0-876d086c5431"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 20:43:25.621289 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.616419    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45915c69-8608-4cb5-a5f0-876d086c5431-kube-api-access-gfwhb" (OuterVolumeSpecName: "kube-api-access-gfwhb") pod "45915c69-8608-4cb5-a5f0-876d086c5431" (UID: "45915c69-8608-4cb5-a5f0-876d086c5431"). InnerVolumeSpecName "kube-api-access-gfwhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:25.641512 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.641485    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:43:25.643697 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.643678    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:43:25.710298 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710251    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5rhq\" (UniqueName: \"kubernetes.io/projected/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-kube-api-access-g5rhq\") pod \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") "
Jul 28 20:43:25.710602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710336    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713fb5ff-30b0-4186-97ad-cb00b0376f0a-plugins-config-volume\") pod \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") "
Jul 28 20:43:25.710602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710402    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-chzz5\" (UniqueName: \"kubernetes.io/projected/713fb5ff-30b0-4186-97ad-cb00b0376f0a-kube-api-access-chzz5\") pod \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\" (UID: \"713fb5ff-30b0-4186-97ad-cb00b0376f0a\") "
Jul 28 20:43:25.710602 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710442    2578 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-plugins-config-volume\") pod \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\" (UID: \"614e919e-06ce-4636-b40a-0cfa3f0cc0c1\") "
Jul 28 20:43:25.710779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710684    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfwhb\" (UniqueName: \"kubernetes.io/projected/45915c69-8608-4cb5-a5f0-876d086c5431-kube-api-access-gfwhb\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.710779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710706    2578 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/45915c69-8608-4cb5-a5f0-876d086c5431-openshift-service-ca\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.710779 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710721    2578 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/45915c69-8608-4cb5-a5f0-876d086c5431-maas-api-tls\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.710944 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710802    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713fb5ff-30b0-4186-97ad-cb00b0376f0a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "713fb5ff-30b0-4186-97ad-cb00b0376f0a" (UID: "713fb5ff-30b0-4186-97ad-cb00b0376f0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:25.710944 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.710817    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "614e919e-06ce-4636-b40a-0cfa3f0cc0c1" (UID: "614e919e-06ce-4636-b40a-0cfa3f0cc0c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 20:43:25.712499 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.712474    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-kube-api-access-g5rhq" (OuterVolumeSpecName: "kube-api-access-g5rhq") pod "614e919e-06ce-4636-b40a-0cfa3f0cc0c1" (UID: "614e919e-06ce-4636-b40a-0cfa3f0cc0c1"). InnerVolumeSpecName "kube-api-access-g5rhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:25.712687 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.712668    2578 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713fb5ff-30b0-4186-97ad-cb00b0376f0a-kube-api-access-chzz5" (OuterVolumeSpecName: "kube-api-access-chzz5") pod "713fb5ff-30b0-4186-97ad-cb00b0376f0a" (UID: "713fb5ff-30b0-4186-97ad-cb00b0376f0a"). InnerVolumeSpecName "kube-api-access-chzz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 20:43:25.811511 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.811472    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713fb5ff-30b0-4186-97ad-cb00b0376f0a-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.811511 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.811505    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chzz5\" (UniqueName: \"kubernetes.io/projected/713fb5ff-30b0-4186-97ad-cb00b0376f0a-kube-api-access-chzz5\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.811511 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.811515    2578 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-plugins-config-volume\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.811741 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.811524    2578 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rhq\" (UniqueName: \"kubernetes.io/projected/614e919e-06ce-4636-b40a-0cfa3f0cc0c1-kube-api-access-g5rhq\") on node \"ip-10-0-133-160.ec2.internal\" DevicePath \"\""
Jul 28 20:43:25.887186 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.887151    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"]
Jul 28 20:43:25.895245 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:25.895211    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-1231ba-56f4f54bb-qlsq9"]
Jul 28 20:43:26.575633 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.575596    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6" event={"ID":"713fb5ff-30b0-4186-97ad-cb00b0376f0a","Type":"ContainerDied","Data":"a3bdc00fcf6d3d0b629001e9663c0111ef2476830a3edeeff616a9489bbc6761"}
Jul 28 20:43:26.576125 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.575662    2578 scope.go:117] "RemoveContainer" containerID="5008d124fefa50087d5e8893278e6b77ca3e7d0a4025d79ddc6f21a9157e7fce"
Jul 28 20:43:26.576125 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.575605    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"
Jul 28 20:43:26.576976 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.576948    2578 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"
Jul 28 20:43:26.577036 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.576983    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8" event={"ID":"614e919e-06ce-4636-b40a-0cfa3f0cc0c1","Type":"ContainerDied","Data":"15d47ce2ba03bc64fd57b35ff33bab0e17e5d5092f22abd799321741d6719c50"}
Jul 28 20:43:26.584917 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.584863    2578 scope.go:117] "RemoveContainer" containerID="61841b8a735931d41a28b1c78488a9d3017e723fe8968192c6752cfa3485d7bd"
Jul 28 20:43:26.612154 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.612117    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"]
Jul 28 20:43:26.619886 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.619856    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-1231ba-795f78959b-528b6"]
Jul 28 20:43:26.635044 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.635002    2578 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"]
Jul 28 20:43:26.638773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:26.638742    2578 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-1231ba-7894d8877d-zvvb8"]
Jul 28 20:43:27.651116 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:27.651081    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45915c69-8608-4cb5-a5f0-876d086c5431" path="/var/lib/kubelet/pods/45915c69-8608-4cb5-a5f0-876d086c5431/volumes"
Jul 28 20:43:27.651521 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:27.651448    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614e919e-06ce-4636-b40a-0cfa3f0cc0c1" path="/var/lib/kubelet/pods/614e919e-06ce-4636-b40a-0cfa3f0cc0c1/volumes"
Jul 28 20:43:27.651773 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:27.651760    2578 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="713fb5ff-30b0-4186-97ad-cb00b0376f0a" path="/var/lib/kubelet/pods/713fb5ff-30b0-4186-97ad-cb00b0376f0a/volumes"
Jul 28 20:43:28.300215 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:28.300174    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-nvnk2"]
Jul 28 20:43:41.368649 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:41.368617    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85967bc8b9-cwvcg_78513af9-1638-4bc7-981b-1a2219c1160c/manager/0.log"
Jul 28 20:43:41.379388 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:41.379354    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85967bc8b9-cwvcg_78513af9-1638-4bc7-981b-1a2219c1160c/copy-manifests/0.log"
Jul 28 20:43:42.125855 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:42.125821    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754480-q46d6_fc6896dc-5b67-47d0-b1d4-a73e104712d2/cleanup/0.log"
Jul 28 20:43:42.243716 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:42.243673    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754495-pvbkb_a625c4ac-004d-42f3-9cbc-384d66aaa1d4/cleanup/0.log"
Jul 28 20:43:43.274688 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.274651    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq_d04bb88d-00c0-46fd-bc1d-0138d0ea3f91/util/0.log"
Jul 28 20:43:43.282422 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.282392    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq_d04bb88d-00c0-46fd-bc1d-0138d0ea3f91/pull/0.log"
Jul 28 20:43:43.293356 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.293325    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq_d04bb88d-00c0-46fd-bc1d-0138d0ea3f91/extract/0.log"
Jul 28 20:43:43.411482 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.411449    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm_5aec8309-bef8-4833-918b-fc5d8f7f14d6/extract/0.log"
Jul 28 20:43:43.421237 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.421213    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm_5aec8309-bef8-4833-918b-fc5d8f7f14d6/util/0.log"
Jul 28 20:43:43.434657 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.434633    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm_5aec8309-bef8-4833-918b-fc5d8f7f14d6/pull/0.log"
Jul 28 20:43:43.565747 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.565671    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd_24f3f95e-89a0-4603-9082-e96fc7f35112/pull/0.log"
Jul 28 20:43:43.577506 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.577479    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd_24f3f95e-89a0-4603-9082-e96fc7f35112/extract/0.log"
Jul 28 20:43:43.589175 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.589149    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd_24f3f95e-89a0-4603-9082-e96fc7f35112/util/0.log"
Jul 28 20:43:43.716375 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.716341    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq_46885940-23b6-4079-8abf-92ce1a09bde1/util/0.log"
Jul 28 20:43:43.729712 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.729679    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq_46885940-23b6-4079-8abf-92ce1a09bde1/pull/0.log"
Jul 28 20:43:43.742435 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:43.742403    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq_46885940-23b6-4079-8abf-92ce1a09bde1/extract/0.log"
Jul 28 20:43:44.596351 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:44.596323    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nvnk2_c2cba7f5-e09b-46ec-8370-70fa21301873/limitador/0.log"
Jul 28 20:43:45.215155 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:45.215120    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jkqdq_836f8910-418b-457e-8e24-18f4976a18bd/discovery/0.log"
Jul 28 20:43:47.075754 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:43:47.075718    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b574f647-594xn_6cb891fa-f2eb-429d-abcd-ee896160e98a/router/0.log"
Jul 28 20:44:23.415477 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:23.415443    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lt7fb_4cad5271-713f-4162-b9ad-00cdc9dd0221/global-pull-secret-syncer/0.log"
Jul 28 20:44:23.517070 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:23.517035    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hh7c9_41ca7edc-a640-4fa8-97db-f7b908e53ed1/konnectivity-agent/0.log"
Jul 28 20:44:23.607005 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:23.606972    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-160.ec2.internal_e70b54208e499ae364c57edb3340d4b8/haproxy/0.log"
Jul 28 20:44:27.853613 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:27.853579    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq_d04bb88d-00c0-46fd-bc1d-0138d0ea3f91/extract/0.log"
Jul 28 20:44:27.893759 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:27.893721    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq_d04bb88d-00c0-46fd-bc1d-0138d0ea3f91/util/0.log"
Jul 28 20:44:27.977324 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:27.977292    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759xdzbq_d04bb88d-00c0-46fd-bc1d-0138d0ea3f91/pull/0.log"
Jul 28 20:44:28.026015 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.025981    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm_5aec8309-bef8-4833-918b-fc5d8f7f14d6/extract/0.log"
Jul 28 20:44:28.079698 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.079614    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm_5aec8309-bef8-4833-918b-fc5d8f7f14d6/util/0.log"
Jul 28 20:44:28.130119 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.130087    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0k84xm_5aec8309-bef8-4833-918b-fc5d8f7f14d6/pull/0.log"
Jul 28 20:44:28.186510 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.186480    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd_24f3f95e-89a0-4603-9082-e96fc7f35112/extract/0.log"
Jul 28 20:44:28.230621 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.230586    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd_24f3f95e-89a0-4603-9082-e96fc7f35112/util/0.log"
Jul 28 20:44:28.277022 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.276988    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed732hhbd_24f3f95e-89a0-4603-9082-e96fc7f35112/pull/0.log"
Jul 28 20:44:28.341654 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.341574    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq_46885940-23b6-4079-8abf-92ce1a09bde1/extract/0.log"
Jul 28 20:44:28.379876 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.379847    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq_46885940-23b6-4079-8abf-92ce1a09bde1/util/0.log"
Jul 28 20:44:28.411799 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:28.411770    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef142lfq_46885940-23b6-4079-8abf-92ce1a09bde1/pull/0.log"
Jul 28 20:44:29.126750 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:29.126714    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-nvnk2_c2cba7f5-e09b-46ec-8370-70fa21301873/limitador/0.log"
Jul 28 20:44:31.086003 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:31.085565    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-j8m4d_f985925d-7690-4f3d-b33d-d12804131a78/cluster-monitoring-operator/0.log"
Jul 28 20:44:31.291654 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:31.291619    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7kqt_fd064301-83bd-4224-a7ce-53552a9543fa/node-exporter/0.log"
Jul 28 20:44:31.321575 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:31.321544    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7kqt_fd064301-83bd-4224-a7ce-53552a9543fa/kube-rbac-proxy/0.log"
Jul 28 20:44:31.349054 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:31.348965    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-l7kqt_fd064301-83bd-4224-a7ce-53552a9543fa/init-textfile/0.log"
Jul 28 20:44:33.046121 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046086    2578 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"]
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046433    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046445    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046453    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ca21b13-8bee-47c0-be50-ccb1702d3fb5" containerName="maas-api"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046460    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ca21b13-8bee-47c0-be50-ccb1702d3fb5" containerName="maas-api"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046469    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" containerName="payload-pre-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046477    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" containerName="payload-pre-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046484    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614e919e-06ce-4636-b40a-0cfa3f0cc0c1" containerName="payload-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046489    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="614e919e-06ce-4636-b40a-0cfa3f0cc0c1" containerName="payload-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046497    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046504    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046513    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="713fb5ff-30b0-4186-97ad-cb00b0376f0a" containerName="payload-pre-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046518    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="713fb5ff-30b0-4186-97ad-cb00b0376f0a" containerName="payload-pre-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046525    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9771615a-66ae-4029-ac44-9251e4ba1793" containerName="payload-pre-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046530    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="9771615a-66ae-4029-ac44-9251e4ba1793" containerName="payload-pre-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046538    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bf465df-f192-461f-a3e7-d45363b6ebea" containerName="payload-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046543    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bf465df-f192-461f-a3e7-d45363b6ebea" containerName="payload-processing"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046551    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy"
Jul 28 20:44:33.046550 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046556    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046579    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" containerName="payload-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046585    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" containerName="payload-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046591    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45915c69-8608-4cb5-a5f0-876d086c5431" containerName="maas-api"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046596    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="45915c69-8608-4cb5-a5f0-876d086c5431" containerName="maas-api"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046605    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1693d8d-6ec8-4a51-a2de-579563c69490" containerName="maas-api"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046610    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1693d8d-6ec8-4a51-a2de-579563c69490" containerName="maas-api"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046616    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4697002-f9b3-49c7-bf9b-ee7d10866031" containerName="payload-pre-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046621    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4697002-f9b3-49c7-bf9b-ee7d10866031" containerName="payload-pre-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046630    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046635    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046643    2578 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046647    2578 state_mem.go:107] "Deleted CPUSet assignment" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046700    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="262167d6-b39d-4b18-af23-68af6cbe3f43" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046707    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="45915c69-8608-4cb5-a5f0-876d086c5431" containerName="maas-api"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046712    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4697002-f9b3-49c7-bf9b-ee7d10866031" containerName="payload-pre-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046718    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046723    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046728    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1693d8d-6ec8-4a51-a2de-579563c69490" containerName="maas-api"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046733    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="713fb5ff-30b0-4186-97ad-cb00b0376f0a" containerName="payload-pre-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046738    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bf465df-f192-461f-a3e7-d45363b6ebea" containerName="payload-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046744    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2a73b7f-7ca5-406a-aeaa-7bcb26cdf39f" containerName="payload-pre-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046749    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="8afe3139-f01f-403f-9b07-0d3a39022dbb" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046753    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="9771615a-66ae-4029-ac44-9251e4ba1793" containerName="payload-pre-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046758    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9ac2830-e487-434a-8041-1758fdf2f255" containerName="istio-proxy"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046763    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc4304ed-f61f-4e90-9c80-d4d294f1e9bb" containerName="payload-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046768    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="614e919e-06ce-4636-b40a-0cfa3f0cc0c1" containerName="payload-processing"
Jul 28 20:44:33.047123 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.046773    2578 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ca21b13-8bee-47c0-be50-ccb1702d3fb5" containerName="maas-api"
Jul 28 20:44:33.049947 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.049924    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.053539 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.053513    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz499\"/\"openshift-service-ca.crt\""
Jul 28 20:44:33.054500 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.054479    2578 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bz499\"/\"kube-root-ca.crt\""
Jul 28 20:44:33.054623 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.054535    2578 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bz499\"/\"default-dockercfg-mdklh\""
Jul 28 20:44:33.065460 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.065431    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"]
Jul 28 20:44:33.199608 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.199570    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-podres\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.199608 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.199612    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-proc\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.199869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.199637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrxb\" (UniqueName: \"kubernetes.io/projected/4a5fe82a-fb62-440b-8419-5e65edb79753-kube-api-access-gxrxb\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.199869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.199710    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-sys\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.199869 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.199798    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-lib-modules\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.300763 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300652    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-lib-modules\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.300763 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300731    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-podres\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.300763 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300754    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-proc\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.301011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrxb\" (UniqueName: \"kubernetes.io/projected/4a5fe82a-fb62-440b-8419-5e65edb79753-kube-api-access-gxrxb\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.301011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300801    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-sys\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.301011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300849    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-lib-modules\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.301011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300864    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-proc\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.301011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300884    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-sys\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.301011 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.300901    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4a5fe82a-fb62-440b-8419-5e65edb79753-podres\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.313184 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.313153    2578 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrxb\" (UniqueName: \"kubernetes.io/projected/4a5fe82a-fb62-440b-8419-5e65edb79753-kube-api-access-gxrxb\") pod \"perf-node-gather-daemonset-jvgrm\" (UID: \"4a5fe82a-fb62-440b-8419-5e65edb79753\") " pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.359220 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.359181    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.385193 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.385164    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-gsjpt_ad0030e8-c33f-49c3-bb42-d0b809f19f98/networking-console-plugin/0.log"
Jul 28 20:44:33.502158 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.502131    2578 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"]
Jul 28 20:44:33.504394 ip-10-0-133-160 kubenswrapper[2578]: W0728 20:44:33.504362    2578 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4a5fe82a_fb62_440b_8419_5e65edb79753.slice/crio-18f4aa3a05a4a4eeca969f45f027b3d19d06df8db55d78dc7a076e1967679db3 WatchSource:0}: Error finding container 18f4aa3a05a4a4eeca969f45f027b3d19d06df8db55d78dc7a076e1967679db3: Status 404 returned error can't find the container with id 18f4aa3a05a4a4eeca969f45f027b3d19d06df8db55d78dc7a076e1967679db3
Jul 28 20:44:33.914402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.914297    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm" event={"ID":"4a5fe82a-fb62-440b-8419-5e65edb79753","Type":"ContainerStarted","Data":"854553c8462120af1a8d12469861dd7463f4883db5c1d3006856ceb57f37e0a1"}
Jul 28 20:44:33.914402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.914365    2578 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm" event={"ID":"4a5fe82a-fb62-440b-8419-5e65edb79753","Type":"ContainerStarted","Data":"18f4aa3a05a4a4eeca969f45f027b3d19d06df8db55d78dc7a076e1967679db3"}
Jul 28 20:44:33.914402 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.914392    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:33.939995 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:33.939937    2578 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm" podStartSLOduration=0.939919675 podStartE2EDuration="939.919675ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 20:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 20:44:33.939671313 +0000 UTC m=+3476.839888157" watchObservedRunningTime="2026-07-28 20:44:33.939919675 +0000 UTC m=+3476.840136516"
Jul 28 20:44:34.064951 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:34.064910    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/1.log"
Jul 28 20:44:34.069673 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:34.069624    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-nnfhr_a270ba75-d93d-4d05-ab56-523a14f234a9/console-operator/2.log"
Jul 28 20:44:34.626910 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:34.626868    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-kpzrq_f18750e0-eeda-45c0-9150-8240779c4755/download-server/0.log"
Jul 28 20:44:35.302003 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:35.301971    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-cfccp_3e30e17e-6f4e-4d92-a1f9-19bdb5b6d0bd/volume-data-source-validator/0.log"
Jul 28 20:44:36.650558 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:36.650528    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jh7xz_fd16eed2-e3aa-4672-bdfc-c9c10b376e9b/dns/0.log"
Jul 28 20:44:36.701314 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:36.701285    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jh7xz_fd16eed2-e3aa-4672-bdfc-c9c10b376e9b/kube-rbac-proxy/0.log"
Jul 28 20:44:37.074846 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:37.074812    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xx69x_e77e8b7e-663f-417e-8e2c-73683e752699/dns-node-resolver/0.log"
Jul 28 20:44:37.727968 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:37.727928    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-974jf_986541f9-ad8a-4755-b6bc-37e010abdbdb/node-ca/0.log"
Jul 28 20:44:39.061966 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:39.061922    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jkqdq_836f8910-418b-457e-8e24-18f4976a18bd/discovery/0.log"
Jul 28 20:44:39.278461 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:39.278432    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b574f647-594xn_6cb891fa-f2eb-429d-abcd-ee896160e98a/router/0.log"
Jul 28 20:44:39.929499 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:39.929470    2578 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bz499/perf-node-gather-daemonset-jvgrm"
Jul 28 20:44:40.173105 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.173065    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6v2wp_33895c8b-396f-428a-98ae-edb4ce1cf0c4/serve-healthcheck-canary/0.log"
Jul 28 20:44:40.375936 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.375902    2578 scope.go:117] "RemoveContainer" containerID="a18aabc9099d273e3cac8c498da83250f2b1344f350d4177d49d4179e01862bb"
Jul 28 20:44:40.799518 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.799483    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-j66nk_3adb22c5-5525-40b6-a40c-42a13c6abd3a/insights-operator/0.log"
Jul 28 20:44:40.800312 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.800288    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-j66nk_3adb22c5-5525-40b6-a40c-42a13c6abd3a/insights-operator/1.log"
Jul 28 20:44:40.909506 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.909470    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9sw5_fec949f3-4c83-40dd-8b20-73eca8c3d8ef/kube-rbac-proxy/0.log"
Jul 28 20:44:40.939687 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.939664    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9sw5_fec949f3-4c83-40dd-8b20-73eca8c3d8ef/exporter/0.log"
Jul 28 20:44:40.967096 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:40.967069    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9sw5_fec949f3-4c83-40dd-8b20-73eca8c3d8ef/extractor/0.log"
Jul 28 20:44:43.326549 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:43.326452    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jkqdq_836f8910-418b-457e-8e24-18f4976a18bd/discovery/0.log"
Jul 28 20:44:43.509411 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:43.509380    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b574f647-594xn_6cb891fa-f2eb-429d-abcd-ee896160e98a/router/0.log"
Jul 28 20:44:44.171200 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:44.171166    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85967bc8b9-cwvcg_78513af9-1638-4bc7-981b-1a2219c1160c/manager/0.log"
Jul 28 20:44:44.230317 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:44.230258    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-85967bc8b9-cwvcg_78513af9-1638-4bc7-981b-1a2219c1160c/copy-manifests/0.log"
Jul 28 20:44:45.127750 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:45.127701    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jkqdq_836f8910-418b-457e-8e24-18f4976a18bd/discovery/0.log"
Jul 28 20:44:45.369670 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:45.369636    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-86b574f647-594xn_6cb891fa-f2eb-429d-abcd-ee896160e98a/router/0.log"
Jul 28 20:44:45.972979 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:45.972896    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6846c7dcc9-fhp67_3b2e7cdc-aeb3-421e-a530-5fec50e87c8b/manager/0.log"
Jul 28 20:44:46.018221 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:46.018182    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-jsn5w_e1e18ad1-0a6a-482b-934f-ba587e980a1f/openshift-lws-operator/0.log"
Jul 28 20:44:51.784617 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:51.784577    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2_0377c7ec-588d-42a2-aed1-e21b9f626411/kube-storage-version-migrator-operator/1.log"
Jul 28 20:44:51.785379 ip-10-0-133-160 kubenswrapper[2578]: I0728 20:44:51.785359    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-5z6b2_0377c7ec-588d-42a2-aed1-e21b9f626411/kube-storage-version-migrator-operator/0.log"