Jul 09 16:04:13.449456 ip-10-0-133-91 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 16:04:13.829464 ip-10-0-133-91 kubenswrapper[2558]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:04:13.829464 ip-10-0-133-91 kubenswrapper[2558]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 16:04:13.829464 ip-10-0-133-91 kubenswrapper[2558]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:04:13.829464 ip-10-0-133-91 kubenswrapper[2558]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 16:04:13.829464 ip-10-0-133-91 kubenswrapper[2558]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:04:13.830257 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.830165    2558 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 16:04:13.834599 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834575    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:04:13.834599 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834601    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834606    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834611    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834616    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834620    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834623    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834626    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834629    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834632    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834634    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834637    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834639    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834642    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834645    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834648    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834650    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834653    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834656    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834658    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:04:13.834690 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834660    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834664    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834666    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834669    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834679    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834682    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834684    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834687    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834690    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834692    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834695    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834698    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834700    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834703    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834708    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834711    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834714    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834717    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834720    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834723    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:04:13.835159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834726    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834728    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834731    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834733    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834736    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834738    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834740    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834743    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834745    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834747    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834750    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834752    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834755    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834757    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834760    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834763    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834766    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834768    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834771    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834773    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:04:13.835641 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834776    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834778    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834781    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834783    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834786    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834788    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834791    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834794    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834796    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834799    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834801    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834805    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834807    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834810    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834812    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834814    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834817    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834820    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834823    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834825    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:04:13.836194 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834828    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834830    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834833    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834836    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834838    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834841    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834844    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834847    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834850    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834852    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834855    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834858    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834860    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834862    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834865    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834867    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834870    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834872    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.834874    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835625    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:04:13.836677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835632    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835636    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835639    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835641    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835645    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835648    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835652    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835664    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835667    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835670    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835673    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835675    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835678    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835680    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835683    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835686    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835688    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835693    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835699    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:04:13.837204 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835703    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835706    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835709    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835712    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835716    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835720    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835723    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835726    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835728    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835732    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835734    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835737    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835740    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835742    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835745    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835747    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835750    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835752    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835756    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:04:13.837686 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835759    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835762    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835764    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835766    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835769    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835771    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835773    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835776    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835779    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835781    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835783    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835786    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835789    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835792    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835796    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835799    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835802    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835804    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835807    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:04:13.838178 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835810    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835813    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835815    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835818    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835820    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835823    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835826    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835828    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835831    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835834    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835836    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835839    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835841    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835844    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835847    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835849    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835852    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835854    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835857    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835859    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:04:13.838639 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835862    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835864    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835867    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835869    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835872    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835875    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835878    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835881    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835883    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835886    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835889    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835891    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835894    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835896    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835899    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835901    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835903    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835906    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835908    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835911    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:04:13.839151 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.835913    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836333    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836338    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836341    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836344    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836347    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836350    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836353    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836355    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836358    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836361    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836363    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836366    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836368    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836371    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836374    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836377    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836380    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836382    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836385    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:04:13.839643 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836388    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836390    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836393    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836396    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836398    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836401    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836403    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836408    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836411    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836414    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836417    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836420    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836422    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836425    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836428    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836430    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836433    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836435    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836446    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:04:13.840142 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836450    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836452    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836455    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836457    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836460    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836462    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836465    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836467    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836470    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836474    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836480    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836483    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836485    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836488    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836491    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836493    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836496    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836498    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836501    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836503    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:04:13.840646 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836505    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836508    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836510    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836513    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836515    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836518    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836520    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836524    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836526    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836529    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836532    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836536    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836547    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836549    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836552    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836555    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836557    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836560    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836562    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:04:13.841154 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836564    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836567    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836569    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836574    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836576    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836579    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836583    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836586    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836589    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836592    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836594    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836597    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836600    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836602    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836605    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836607    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836610    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836612    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836615    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836617    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:04:13.841627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836620    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.836622    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836710    2558 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836718    2558 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836724    2558 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836733    2558 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836753    2558 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836761    2558 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836767    2558 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836772    2558 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836775    2558 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836778    2558 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836786    2558 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836789    2558 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836792    2558 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836796    2558 flags.go:64] FLAG: --cgroup-root=""
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836799    2558 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836802    2558 flags.go:64] FLAG: --client-ca-file=""
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836815    2558 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836818    2558 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836825    2558 flags.go:64] FLAG: --cluster-domain=""
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836828    2558 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836830    2558 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836834    2558 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 16:04:13.842123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836837    2558 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836841    2558 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836844    2558 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836847    2558 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836850    2558 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836853    2558 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836856    2558 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836859    2558 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836862    2558 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836865    2558 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836870    2558 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836873    2558 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836875    2558 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836880    2558 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836883    2558 flags.go:64] FLAG: --enable-server="true"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836892    2558 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836898    2558 flags.go:64] FLAG: --event-burst="100"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836901    2558 flags.go:64] FLAG: --event-qps="50"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836904    2558 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836907    2558 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836910    2558 flags.go:64] FLAG: --eviction-hard=""
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836914    2558 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836916    2558 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836920    2558 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836923    2558 flags.go:64] FLAG: --eviction-soft=""
Jul 09 16:04:13.842692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836926    2558 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836929    2558 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836932    2558 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836935    2558 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836938    2558 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836940    2558 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836943    2558 flags.go:64] FLAG: --feature-gates=""
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836947    2558 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836950    2558 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836953    2558 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836957    2558 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836960    2558 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836963    2558 flags.go:64] FLAG: --help="false"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836966    2558 flags.go:64] FLAG: --hostname-override="ip-10-0-133-91.ec2.internal"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836969    2558 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836972    2558 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836975    2558 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836978    2558 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836981    2558 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836984    2558 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836989    2558 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836991    2558 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836994    2558 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 16:04:13.843305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.836999    2558 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837002    2558 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837005    2558 flags.go:64] FLAG: --kube-reserved=""
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837008    2558 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837011    2558 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837014    2558 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837017    2558 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837020    2558 flags.go:64] FLAG: --lock-file=""
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837023    2558 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837026    2558 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837030    2558 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837035    2558 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837038    2558 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837041    2558 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837044    2558 flags.go:64] FLAG: --logging-format="text"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837047    2558 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837051    2558 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837054    2558 flags.go:64] FLAG: --manifest-url=""
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837057    2558 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837061    2558 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837064    2558 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837069    2558 flags.go:64] FLAG: --max-pods="110"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837072    2558 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837075    2558 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837077    2558 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 16:04:13.843897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837080    2558 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837101    2558 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837106    2558 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837109    2558 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837121    2558 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837124    2558 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837127    2558 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837130    2558 flags.go:64] FLAG: --pod-cidr=""
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837134    2558 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837141    2558 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837144    2558 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837147    2558 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837150    2558 flags.go:64] FLAG: --port="10250"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837153    2558 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837156    2558 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00c2c744a971c3f36"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837160    2558 flags.go:64] FLAG: --qos-reserved=""
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837163    2558 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837165    2558 flags.go:64] FLAG: --register-node="true"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837168    2558 flags.go:64] FLAG: --register-with-taints=""
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837177    2558 flags.go:64] FLAG: --registry-burst="10"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837180    2558 flags.go:64] FLAG: --registry-qps="5"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837182    2558 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837185    2558 flags.go:64] FLAG: --reserved-memory=""
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837189    2558 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837192    2558 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 16:04:13.844569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837195    2558 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837198    2558 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837200    2558 flags.go:64] FLAG: --runonce="false"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837203    2558 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837206    2558 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837209    2558 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837212    2558 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837215    2558 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837218    2558 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837221    2558 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837224    2558 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837229    2558 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837232    2558 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837235    2558 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837238    2558 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837241    2558 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837245    2558 flags.go:64] FLAG: --system-cgroups=""
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837248    2558 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837254    2558 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837256    2558 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837259    2558 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837264    2558 flags.go:64] FLAG: --tls-min-version=""
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837268    2558 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837271    2558 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837274    2558 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 16:04:13.845303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837277    2558 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837280    2558 flags.go:64] FLAG: --v="2"
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837285    2558 flags.go:64] FLAG: --version="false"
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837289    2558 flags.go:64] FLAG: --vmodule=""
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837292    2558 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.837296    2558 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837421    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837426    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837429    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837432    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837435    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837438    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837440    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837443    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837445    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837447    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837450    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837452    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837457    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837459    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837462    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:04:13.845927 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837465    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837468    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837470    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837473    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837476    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837478    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837481    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837483    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837488    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837490    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837493    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837495    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837498    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837500    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837503    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837506    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837508    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837511    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837513    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837516    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:04:13.846457 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837518    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837521    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837523    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837526    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837529    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837531    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837536    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837539    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837541    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837546    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837548    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837551    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837553    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837556    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837558    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837562    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837567    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837570    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837572    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:04:13.846943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837575    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837580    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837583    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837586    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837588    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837591    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837594    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837596    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837599    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837601    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837604    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837606    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837609    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837611    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837614    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837616    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837619    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837621    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837624    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837626    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:04:13.847429 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837628    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837631    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837635    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837638    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837640    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837643    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837645    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837648    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837650    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837653    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837656    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837659    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837661    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837665    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837668    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837672    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837676    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837678    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837681    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837683    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:04:13.847934 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837686    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837689    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837692    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837695    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.837697    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.838389    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.846311    2558 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.846340    2558 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.846364    2558 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.846371    2558 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846449    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846456    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846459    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846462    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846466    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:04:13.848436 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846469    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846472    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846475    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846479    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846482    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846486    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846491    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846497    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846500    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846502    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846505    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846508    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846510    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846512    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846515    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846518    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846520    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846522    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846525    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846527    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:04:13.848815 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846530    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846532    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846535    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846538    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846549    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846552    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846554    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846557    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846559    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846564    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846568    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846572    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846576    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846579    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846581    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846584    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846586    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846589    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846591    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846594    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:04:13.849391 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846596    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846598    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846601    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846603    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846606    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846608    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846611    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846613    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846615    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846618    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846621    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846623    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846626    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846628    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846631    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846633    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846637    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846648    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846655    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:04:13.849921 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846660    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846662    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846665    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846667    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846671    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846675    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846678    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846681    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846684    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846686    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846689    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846691    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846695    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846699    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846702    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846705    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846708    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846711    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846714    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:04:13.850405 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846716    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846719    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846724    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846728    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846732    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846736    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846738    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846741    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846743    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846746    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846749    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846752    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846761    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846764    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846767    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:04:13.850882 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846769    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.846775    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846944    2558 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846952    2558 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846957    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846962    2558 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846974    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846977    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846981    2558 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846986    2558 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846988    2558 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846991    2558 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846994    2558 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846997    2558 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.846999    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:04:13.851272 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847002    2558 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847004    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847007    2558 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847009    2558 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847012    2558 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847015    2558 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847017    2558 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847020    2558 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847022    2558 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847025    2558 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847027    2558 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847030    2558 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847033    2558 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847039    2558 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847043    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847053    2558 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847056    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847059    2558 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847062    2558 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847064    2558 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:04:13.851651 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847066    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847070    2558 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847074    2558 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847077    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847080    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847082    2558 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847101    2558 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847106    2558 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847110    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847113    2558 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847115    2558 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847118    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847121    2558 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847123    2558 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847126    2558 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847129    2558 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847131    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847134    2558 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847136    2558 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:04:13.852160 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847138    2558 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847141    2558 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847143    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847146    2558 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847149    2558 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847152    2558 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847155    2558 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847157    2558 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847160    2558 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847170    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847174    2558 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847179    2558 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847183    2558 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847187    2558 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847190    2558 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847192    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847195    2558 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847198    2558 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847201    2558 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847204    2558 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:04:13.852621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847206    2558 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847209    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847211    2558 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847214    2558 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847216    2558 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847218    2558 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847221    2558 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847223    2558 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847225    2558 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847228    2558 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847230    2558 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847233    2558 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847235    2558 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847238    2558 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847240    2558 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847243    2558 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847247    2558 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847251    2558 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847260    2558 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:04:13.853273 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847265    2558 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847268    2558 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847271    2558 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847280    2558 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847283    2558 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847286    2558 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847288    2558 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:13.847290    2558 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.847296    2558 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.847486    2558 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.849876    2558 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.851600    2558 server.go:1019] "Starting client certificate rotation"
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.851711    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 16:04:13.853776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.851751    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 16:04:13.856525 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.856466    2558 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 16:04:13.860760 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.860731    2558 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 16:04:13.870902 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.870878    2558 log.go:25] "Validated CRI v1 runtime API"
Jul 09 16:04:13.873321 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.873304    2558 log.go:25] "Validated CRI v1 image API"
Jul 09 16:04:13.874292 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.874273    2558 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 16:04:13.879582 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.879556    2558 fs.go:135] Filesystem UUIDs: map[2765762b-16a8-48d0-866a-94b8c18de871:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c1e1d663-5a93-4fb0-afe4-38369f3f3eca:/dev/nvme0n1p4]
Jul 09 16:04:13.879714 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.879580    2558 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 16:04:13.884684 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.884659    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 16:04:13.885326 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.885204    2558 manager.go:216] Machine: {Timestamp:2026-07-09 16:04:13.883855565 +0000 UTC m=+0.318462606 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3092209 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ca84f75d643e0a11ddc3adf747dc2 SystemUUID:ec2ca84f-75d6-43e0-a11d-dc3adf747dc2 BootID:d27786c3-e4cf-4ae3-8862-fb992cf3e468 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:63:d1:f4:d2:17 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:63:d1:f4:d2:17 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:52:7c:e0:c3:5b:90 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 16:04:13.885326 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.885322    2558 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 16:04:13.885440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.885428    2558 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 16:04:13.887628 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.887595    2558 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 16:04:13.887829 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.887635    2558 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-91.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 16:04:13.888538 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.888527    2558 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 16:04:13.888578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.888541    2558 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 16:04:13.888578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.888549    2558 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 16:04:13.888578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.888568    2558 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 16:04:13.889848 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.889836    2558 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 16:04:13.893703 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.893683    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 16:04:13.894040 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894028    2558 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 16:04:13.894077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894066    2558 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 16:04:13.894130 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894078    2558 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 16:04:13.894130 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894114    2558 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 16:04:13.894130 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894124    2558 kubelet.go:398] "Adding apiserver pod source"
Jul 09 16:04:13.894211 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894134    2558 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 16:04:13.894581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894567    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 16:04:13.894635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.894586    2558 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 16:04:13.897823 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.897683    2558 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 16:04:13.899649 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.899629    2558 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 16:04:13.899735 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.899672    2558 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 16:04:13.900466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900451    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 16:04:13.900504 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900472    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 16:04:13.900504 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900479    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 16:04:13.900504 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900485    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 16:04:13.900504 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900494    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 16:04:13.900504 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900500    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 16:04:13.900645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900509    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 16:04:13.900645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900514    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 16:04:13.900645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900521    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 16:04:13.900645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900528    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 16:04:13.900645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900536    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 16:04:13.900645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.900542    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 16:04:13.901710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.901697    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 16:04:13.901710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.901708    2558 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 16:04:13.904240 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.904214    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjbkc"
Jul 09 16:04:13.905001 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.904977    2558 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-91.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 16:04:13.905121 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.905078    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 16:04:13.905172 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.905124    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-133-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 16:04:13.905300 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.905281    2558 server.go:1268] "Started kubelet"
Jul 09 16:04:13.905393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.905333    2558 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 16:04:13.906126 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.906057    2558 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 16:04:13.906198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.906155    2558 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 16:04:13.906319 ip-10-0-133-91 systemd[1]: Started Kubernetes Kubelet.
Jul 09 16:04:13.907396 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.907379    2558 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 16:04:13.907578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.907563    2558 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 16:04:13.912184 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.912147    2558 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 16:04:13.912409 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.912395    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-wjbkc"
Jul 09 16:04:13.912858 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.911908    2558 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-91.ec2.internal.18c0aa9f5ec5abc4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-91.ec2.internal,UID:ip-10-0-133-91.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-91.ec2.internal,},FirstTimestamp:2026-07-09 16:04:13.905251268 +0000 UTC m=+0.339858309,LastTimestamp:2026-07-09 16:04:13.905251268 +0000 UTC m=+0.339858309,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-91.ec2.internal,}"
Jul 09 16:04:13.914540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.914513    2558 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 16:04:13.914540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.914525    2558 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 16:04:13.914711 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.914587    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:13.915178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915126    2558 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 16:04:13.915178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915129    2558 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 16:04:13.915178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915151    2558 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 16:04:13.915371 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915251    2558 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 16:04:13.915371 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915260    2558 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 16:04:13.915832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915813    2558 factory.go:153] Registering CRI-O factory
Jul 09 16:04:13.915913 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915837    2558 factory.go:223] Registration of the crio container factory successfully
Jul 09 16:04:13.915964 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915914    2558 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 16:04:13.915964 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915924    2558 factory.go:55] Registering systemd factory
Jul 09 16:04:13.915964 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915932    2558 factory.go:223] Registration of the systemd container factory successfully
Jul 09 16:04:13.915964 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915954    2558 factory.go:103] Registering Raw factory
Jul 09 16:04:13.916136 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.915968    2558 manager.go:1196] Started watching for new ooms in manager
Jul 09 16:04:13.916309 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.916299    2558 manager.go:349] Starting recovery of all containers
Jul 09 16:04:13.917882 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.917857    2558 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-133-91.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 16:04:13.917989 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.917971    2558 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 16:04:13.928251 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.928043    2558 manager.go:354] Recovery completed
Jul 09 16:04:13.930102 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.929954    2558 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 09 16:04:13.933234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.933217    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:13.935666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.935641    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:13.935748 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.935683    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:13.935748 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.935694    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:13.936296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.936278    2558 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 16:04:13.936296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.936291    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:13.936493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.936292    2558 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 16:04:13.936493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.936336    2558 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 16:04:13.937030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.937011    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:13.937140 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.937042    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:13.937140 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.937052    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:13.937833 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.937819    2558 policy_none.go:49] "None policy: Start"
Jul 09 16:04:13.937893 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.937841    2558 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 16:04:13.937893 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.937854    2558 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 16:04:13.939048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.939035    2558 policy_none.go:47] "Start"
Jul 09 16:04:13.981891 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.981868    2558 manager.go:337] "Starting Device Plugin manager"
Jul 09 16:04:13.982034 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.981926    2558 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 16:04:13.982034 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.981939    2558 server.go:87] "Starting device plugin registration server"
Jul 09 16:04:13.982281 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.982267    2558 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 16:04:13.982390 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.982284    2558 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 16:04:13.982390 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.982358    2558 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 16:04:13.982634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.982450    2558 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 16:04:13.982634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:13.982459    2558 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 16:04:13.983116 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.983071    2558 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 16:04:13.983196 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:13.983157    2558 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:14.082492 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.082388    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:14.083647 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.083637    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:14.083731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.083681    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:14.083731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.083696    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:14.083731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.083724    2558 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.093400 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.093365    2558 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.093400 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.093399    2558 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-133-91.ec2.internal\": node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:14.110334 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.110306    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:14.117235 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.117181    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 16:04:14.118688 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.118666    2558 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 16:04:14.118841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.118709    2558 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 16:04:14.118841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.118730    2558 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 16:04:14.118841 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.118764    2558 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 16:04:14.121193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.121170    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:04:14.211184 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.211147    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:14.219638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.219607    2558 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"]
Jul 09 16:04:14.219740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.219701    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:14.220788 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.220767    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:14.220948 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.220800    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:14.220948 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.220811    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:14.222388 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.222373    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:14.222502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.222486    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.222558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.222528    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:14.223207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.223191    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:14.223207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.223198    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:14.223334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.223218    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:14.223334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.223224    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:14.223334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.223235    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:14.223334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.223240    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:14.224440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.224427    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.224491 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.224456    2558 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:04:14.225198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.225182    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:04:14.225290 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.225208    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:04:14.225290 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.225217    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:04:14.241781 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.241757    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-91.ec2.internal\" not found" node="ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.245538 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.245519    2558 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-91.ec2.internal\" not found" node="ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.312139 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.312083    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:14.318222 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.318196    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2a43db58e6bc067f1a769598e52ae5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal\" (UID: \"e2a43db58e6bc067f1a769598e52ae5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.318305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.318227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a43db58e6bc067f1a769598e52ae5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal\" (UID: \"e2a43db58e6bc067f1a769598e52ae5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.318305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.318246    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0964284bd0ac4fa0d88f69cd97267984-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.ec2.internal\" (UID: \"0964284bd0ac4fa0d88f69cd97267984\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.412735 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:14.412654    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:14.419124 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.419074    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2a43db58e6bc067f1a769598e52ae5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal\" (UID: \"e2a43db58e6bc067f1a769598e52ae5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.419222 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.419156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a43db58e6bc067f1a769598e52ae5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal\" (UID: \"e2a43db58e6bc067f1a769598e52ae5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.419222 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.419173    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e2a43db58e6bc067f1a769598e52ae5e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal\" (UID: \"e2a43db58e6bc067f1a769598e52ae5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.419222 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.419186    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0964284bd0ac4fa0d88f69cd97267984-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.ec2.internal\" (UID: \"0964284bd0ac4fa0d88f69cd97267984\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.419334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.419223    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0964284bd0ac4fa0d88f69cd97267984-config\") pod \"kube-apiserver-proxy-ip-10-0-133-91.ec2.internal\" (UID: \"0964284bd0ac4fa0d88f69cd97267984\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:14.419334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:14.419225    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e2a43db58e6bc067f1a769598e52ae5e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal\" (UID: \"e2a43db58e6bc067f1a769598e52ae5e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:15.566068 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:15.566032    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:15.597530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:15.597499    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:15.602058 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:15.601742    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:15.667065 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:15.667021    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:15.767558 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:15.767488    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:15.868033 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:15.867995    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:15.903362 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:15.903333    2558 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 16:04:15.904091 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:15.903527    2558 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 16:04:15.969039 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:15.969001    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:15.970792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:15.970752    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 15:59:14 +0000 UTC" deadline="2028-01-30 00:49:08.691222913 +0000 UTC"
Jul 09 16:04:15.970851 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:15.970793    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13664h44m52.720433187s"
Jul 09 16:04:16.069974 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:16.069937    2558 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-133-91.ec2.internal\" not found"
Jul 09 16:04:16.099146 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.099116    2558 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:04:16.154101 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.154077    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:04:16.167804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.167772    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:16.170092 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.170064    2558 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:04:16.174841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.174736    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal" event={"ID":"e2a43db58e6bc067f1a769598e52ae5e","Type":"ContainerStarted","Data":"9023d61b0c0d0e7a26fc2cbeb92bf3081d0ad570dd141d65d1686660e9990c5c"}
Jul 09 16:04:16.175776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.175751    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal" event={"ID":"0964284bd0ac4fa0d88f69cd97267984","Type":"ContainerStarted","Data":"48b43c96ad9acf5cdb5f97ccd342e49208d0cd167c93fb8385d61a8cc6286dcf"}
Jul 09 16:04:16.183215 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.183192    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 16:04:16.184234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.184217    2558 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal"
Jul 09 16:04:16.194007 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.193984    2558 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 16:04:16.274375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.274347    2558 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:04:16.948557 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.948524    2558 apiserver.go:52] "Watching apiserver"
Jul 09 16:04:16.956745 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.956724    2558 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:04:16.957853 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.957830    2558 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 16:04:16.959900 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.959863    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-x9k9w","kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xsrbm","openshift-dns/node-resolver-stdmx","openshift-image-registry/node-ca-44htx","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal","openshift-multus/multus-m5g2d","openshift-multus/network-metrics-daemon-6wzgz","kube-system/konnectivity-agent-hhw4n","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l","openshift-multus/multus-additional-cni-plugins-2gqn8","openshift-network-diagnostics/network-check-target-njhlg","openshift-network-operator/iptables-alerter-b78r9","openshift-ovn-kubernetes/ovnkube-node-px6rt"]
Jul 09 16:04:16.962229 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.962210    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.963524 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.963499    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:16.964751 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.964729    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.965248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.965223    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wtrck\""
Jul 09 16:04:16.965361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.965281    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 16:04:16.965361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.965292    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.965571 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.965555    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 16:04:16.965638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.965570    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.965830 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.965815    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:16.966449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.966434    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.966688 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.966673    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4lq4q\""
Jul 09 16:04:16.966753 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.966689    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 16:04:16.966817 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.966804    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.967422 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.967405    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:16.968513 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.968098    2558 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 16:04:16.970204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.968638    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vndj7\""
Jul 09 16:04:16.970204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.969139    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.970204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970023    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.970655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970434    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.970655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970467    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 16:04:16.970655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970486    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.970799 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970788    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.970850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970805    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-85ldd\""
Jul 09 16:04:16.970850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.970838    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.971254 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.971235    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.971649 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.971627    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rpr69\""
Jul 09 16:04:16.972078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.972059    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:16.974213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.973976    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:16.974213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.973978    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.974213 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:16.974049    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:16.974213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.974091    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mxkhb\""
Jul 09 16:04:16.974213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.974142    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 16:04:16.975037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.975021    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 16:04:16.975300 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.975280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:16.975378 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:16.975358    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:16.975415 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.975388    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 16:04:16.975415 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.975388    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.976257 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.976242    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 16:04:16.976755 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.976736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:16.978246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978212    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-jzws2\""
Jul 09 16:04:16.978246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978228    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 16:04:16.978381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978214    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 16:04:16.978381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978325    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-run\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.978381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978371    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-system-cni-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.978520 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978456    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-var-lib-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.978520 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978497    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.978615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-k8s-cni-cncf-io\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.978615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978566    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-cni-multus\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.978615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978583    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysctl-d\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.978615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978599    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-run-ovn-kubernetes\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978649    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-cni-bin\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978687    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-daemon-config\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-sys\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978734    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-serviceca\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978762    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-etc-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978777    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:16.978797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978785    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-log-socket\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-var-lib-kubelet\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978829    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04b0d083-c5d4-4cc0-906d-c373319e1c97-hosts-file\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-host\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978873    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-ovnkube-config\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978928    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-modprobe-d\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978951    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctjnb\" (UniqueName: \"kubernetes.io/projected/06fe573a-c004-4886-add9-ab05d9a0e8e7-kube-api-access-ctjnb\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.978976    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-slash\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-cni-bin\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979043    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-cni-netd\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysctl-conf\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979087    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysconfig\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979140    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-lib-modules\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-run-netns\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979200    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-env-overrides\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979226    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zlv\" (UniqueName: \"kubernetes.io/projected/41b54e70-0b93-411a-8cbb-6e4f68209e6a-kube-api-access-j4zlv\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979252    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-host\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979306    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0d083-c5d4-4cc0-906d-c373319e1c97-tmp-dir\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979354    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr8n9\" (UniqueName: \"kubernetes.io/projected/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-kube-api-access-wr8n9\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979383    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-etc-kubernetes\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979405    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-tuned\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-systemd-units\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979449    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979487    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-ovnkube-script-lib\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979520    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-cni-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-cni-binary-copy\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979567    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979568    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979568    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-netns\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979615    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41b54e70-0b93-411a-8cbb-6e4f68209e6a-iptables-alerter-script\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979643    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-node-log\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979666    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-socket-dir-parent\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-hostroot\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979710    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41b54e70-0b93-411a-8cbb-6e4f68209e6a-host-slash\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979741    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-kubernetes\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979767    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-cnibin\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979789    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-os-release\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979810    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-conf-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-multus-certs\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.979860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979856    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-kubelet\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979898    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d30497fb-b5e7-45da-af81-0ff51251b144-ovn-node-metrics-cert\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnhw6\" (UniqueName: \"kubernetes.io/projected/d30497fb-b5e7-45da-af81-0ff51251b144-kube-api-access-cnhw6\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979947    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-kubelet\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979966    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.979969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsk8q\" (UniqueName: \"kubernetes.io/projected/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-kube-api-access-dsk8q\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-systemd\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980049    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t6nv2\""
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fe573a-c004-4886-add9-ab05d9a0e8e7-tmp\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt6g7\" (UniqueName: \"kubernetes.io/projected/04b0d083-c5d4-4cc0-906d-c373319e1c97-kube-api-access-kt6g7\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980112    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-systemd\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980135    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-ovn\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.980327    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:16.980500 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:16.980387    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:16.981472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.981458    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 16:04:16.981591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.981485    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hk58h\""
Jul 09 16:04:16.981591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.981499    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 16:04:16.985955 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:16.985925    2558 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 16:04:17.017041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.017014    2558 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-6j9r7"
Jul 09 16:04:17.029508 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.029481    2558 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6j9r7"
Jul 09 16:04:17.069695 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.069665    2558 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 16:04:17.080425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080387    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-cni-netd\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.080425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080432    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080461    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysctl-conf\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysconfig\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-lib-modules\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080521    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-run-netns\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-env-overrides\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080562    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080584    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zlv\" (UniqueName: \"kubernetes.io/projected/41b54e70-0b93-411a-8cbb-6e4f68209e6a-kube-api-access-j4zlv\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080604    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080628    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-host\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.080648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080645    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0d083-c5d4-4cc0-906d-c373319e1c97-tmp-dir\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080669    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr8n9\" (UniqueName: \"kubernetes.io/projected/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-kube-api-access-wr8n9\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080694    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080712    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-etc-kubernetes\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080729    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-tuned\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080754    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-systemd-units\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080807    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-ovnkube-script-lib\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080830    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a539fd7d-b1a9-4708-a562-1dfe76c9dc0f-agent-certs\") pod \"konnectivity-agent-hhw4n\" (UID: \"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f\") " pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.081054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.080855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xvx5\" (UniqueName: \"kubernetes.io/projected/0cca3ed8-8427-4ca7-a1af-7541a27782ff-kube-api-access-5xvx5\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081127    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysctl-conf\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081201    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysconfig\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081209    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-systemd-units\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081254    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081269    2558 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081300    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-run-netns\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081450    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-ovnkube-script-lib\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081527    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-etc-kubernetes\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081570    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7b4\" (UniqueName: \"kubernetes.io/projected/58aa49f7-f574-4eca-a269-49754a518cc1-kube-api-access-8h7b4\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081578    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-lib-modules\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081596    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-cni-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-cni-netd\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081626    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-cni-binary-copy\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081651    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-netns\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081674    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41b54e70-0b93-411a-8cbb-6e4f68209e6a-iptables-alerter-script\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-cni-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-node-log\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-netns\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-node-log\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.081776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a539fd7d-b1a9-4708-a562-1dfe76c9dc0f-konnectivity-ca\") pod \"konnectivity-agent-hhw4n\" (UID: \"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f\") " pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b0d083-c5d4-4cc0-906d-c373319e1c97-tmp-dir\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081806    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-system-cni-dir\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081784    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-host\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081832    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081856    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-socket-dir-parent\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081901    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-hostroot\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081926    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41b54e70-0b93-411a-8cbb-6e4f68209e6a-host-slash\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081942    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-socket-dir-parent\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-kubernetes\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081950    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-env-overrides\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081980    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.081994    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/41b54e70-0b93-411a-8cbb-6e4f68209e6a-host-slash\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-kubernetes\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082007    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-cnibin\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082071    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-hostroot\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-cnibin\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.082376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082098    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-os-release\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082125    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-conf-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082153    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-multus-certs\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082179    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-kubelet\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d30497fb-b5e7-45da-af81-0ff51251b144-ovn-node-metrics-cert\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082230    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnhw6\" (UniqueName: \"kubernetes.io/projected/d30497fb-b5e7-45da-af81-0ff51251b144-kube-api-access-cnhw6\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-cni-binary-copy\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082255    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-kubelet\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082281    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsk8q\" (UniqueName: \"kubernetes.io/projected/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-kube-api-access-dsk8q\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082284    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/41b54e70-0b93-411a-8cbb-6e4f68209e6a-iptables-alerter-script\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-systemd\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082329    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fe573a-c004-4886-add9-ab05d9a0e8e7-tmp\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082353    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt6g7\" (UniqueName: \"kubernetes.io/projected/04b0d083-c5d4-4cc0-906d-c373319e1c97-kube-api-access-kt6g7\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082376    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-systemd\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082397    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-ovn\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082203    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-conf-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082426    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082459    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-run\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.083175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-system-cni-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082489    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-systemd\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082507    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-var-lib-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082516    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-kubelet\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082529    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082571    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-systemd\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082577    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082578    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-ovn\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082358    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-os-release\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-k8s-cni-cncf-io\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082607    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-var-lib-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082648    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-run-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082653    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-multus-certs\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082664    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-run-k8s-cni-cncf-io\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-system-cni-dir\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082701    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-kubelet\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082745    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-run\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.083768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082776    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-cni-multus\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysctl-d\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082835    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-run-ovn-kubernetes\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082895    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-cni-multus\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082902    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082946    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-run-ovn-kubernetes\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-os-release\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.082997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-sysctl-d\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083090    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed33335f-5aec-4360-9937-e478cd9fbe04-kubelet-config\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083117    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-device-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083144    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-cni-bin\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083168    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-daemon-config\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-sys\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083198    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-host-var-lib-cni-bin\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083215    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-serviceca\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083243    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed33335f-5aec-4360-9937-e478cd9fbe04-dbus\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083267    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7n8\" (UniqueName: \"kubernetes.io/projected/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-kube-api-access-zw7n8\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.084423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083294    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-etc-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-log-socket\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cnibin\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-var-lib-kubelet\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083417    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04b0d083-c5d4-4cc0-906d-c373319e1c97-hosts-file\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083442    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-host\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-ovnkube-config\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-modprobe-d\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctjnb\" (UniqueName: \"kubernetes.io/projected/06fe573a-c004-4886-add9-ab05d9a0e8e7-kube-api-access-ctjnb\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083539    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-log-socket\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083540    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-slash\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083580    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-cni-bin\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-slash\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083655    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-multus-daemon-config\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083661    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-host\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083719    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-sys\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-modprobe-d\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.085207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-etc-openvswitch\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.086083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d30497fb-b5e7-45da-af81-0ff51251b144-host-cni-bin\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.086083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083843    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fe573a-c004-4886-add9-ab05d9a0e8e7-var-lib-kubelet\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.086083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.083936    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/04b0d083-c5d4-4cc0-906d-c373319e1c97-hosts-file\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.086083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.084214    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-serviceca\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.086083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.084739    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d30497fb-b5e7-45da-af81-0ff51251b144-ovnkube-config\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.086083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.085194    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/06fe573a-c004-4886-add9-ab05d9a0e8e7-etc-tuned\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.086703 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.086683    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06fe573a-c004-4886-add9-ab05d9a0e8e7-tmp\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.087866 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.087843    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d30497fb-b5e7-45da-af81-0ff51251b144-ovn-node-metrics-cert\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.098372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.098335    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr8n9\" (UniqueName: \"kubernetes.io/projected/bd8a93ba-1327-414e-9755-4c2d0ab8a9d8-kube-api-access-wr8n9\") pod \"node-ca-44htx\" (UID: \"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8\") " pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.099416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.099383    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsk8q\" (UniqueName: \"kubernetes.io/projected/6dc1043f-8908-4897-aaa0-a94fc42d5ef5-kube-api-access-dsk8q\") pod \"multus-m5g2d\" (UID: \"6dc1043f-8908-4897-aaa0-a94fc42d5ef5\") " pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.099572 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.099388    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt6g7\" (UniqueName: \"kubernetes.io/projected/04b0d083-c5d4-4cc0-906d-c373319e1c97-kube-api-access-kt6g7\") pod \"node-resolver-stdmx\" (UID: \"04b0d083-c5d4-4cc0-906d-c373319e1c97\") " pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.099633 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.099599    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zlv\" (UniqueName: \"kubernetes.io/projected/41b54e70-0b93-411a-8cbb-6e4f68209e6a-kube-api-access-j4zlv\") pod \"iptables-alerter-b78r9\" (UID: \"41b54e70-0b93-411a-8cbb-6e4f68209e6a\") " pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.101803 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.101217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnhw6\" (UniqueName: \"kubernetes.io/projected/d30497fb-b5e7-45da-af81-0ff51251b144-kube-api-access-cnhw6\") pod \"ovnkube-node-px6rt\" (UID: \"d30497fb-b5e7-45da-af81-0ff51251b144\") " pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.104947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.104911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctjnb\" (UniqueName: \"kubernetes.io/projected/06fe573a-c004-4886-add9-ab05d9a0e8e7-kube-api-access-ctjnb\") pod \"tuned-xsrbm\" (UID: \"06fe573a-c004-4886-add9-ab05d9a0e8e7\") " pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.184496 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184515    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184546    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a539fd7d-b1a9-4708-a562-1dfe76c9dc0f-agent-certs\") pod \"konnectivity-agent-hhw4n\" (UID: \"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f\") " pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xvx5\" (UniqueName: \"kubernetes.io/projected/0cca3ed8-8427-4ca7-a1af-7541a27782ff-kube-api-access-5xvx5\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184597    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7b4\" (UniqueName: \"kubernetes.io/projected/58aa49f7-f574-4eca-a269-49754a518cc1-kube-api-access-8h7b4\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-etc-selinux\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184624    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a539fd7d-b1a9-4708-a562-1dfe76c9dc0f-konnectivity-ca\") pod \"konnectivity-agent-hhw4n\" (UID: \"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f\") " pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184677    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-system-cni-dir\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.184699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184695    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184733    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184758    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184777    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184793    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184812    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184828    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-os-release\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed33335f-5aec-4360-9937-e478cd9fbe04-kubelet-config\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184860    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-device-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184897    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed33335f-5aec-4360-9937-e478cd9fbe04-dbus\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184913    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7n8\" (UniqueName: \"kubernetes.io/projected/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-kube-api-access-zw7n8\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184941    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cnibin\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184959    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.184982    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.185113 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.185100    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:17.185662 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185125    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a539fd7d-b1a9-4708-a562-1dfe76c9dc0f-konnectivity-ca\") pod \"konnectivity-agent-hhw4n\" (UID: \"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f\") " pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.185662 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.185153    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:17.685137555 +0000 UTC m=+3.066954321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:17.185662 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185184    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-system-cni-dir\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185662 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185800 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.185800 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-registration-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.185994 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185972    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-device-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.186060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.185990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cnibin\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.186060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-sys-fs\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.186060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.186060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186042    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-os-release\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.186246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186035    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ed33335f-5aec-4360-9937-e478cd9fbe04-kubelet-config\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.186246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186143    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/58aa49f7-f574-4eca-a269-49754a518cc1-socket-dir\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.186246 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.186144    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:17.186246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186155    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ed33335f-5aec-4360-9937-e478cd9fbe04-dbus\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.186246 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.186197    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:04:17.686184314 +0000 UTC m=+3.068001081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:17.186524 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186499    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.186631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.186614    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-cni-binary-copy\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.188071 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.188047    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a539fd7d-b1a9-4708-a562-1dfe76c9dc0f-agent-certs\") pod \"konnectivity-agent-hhw4n\" (UID: \"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f\") " pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.194986 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.194947    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:17.194986 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.194988    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:17.195228 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.195003    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:17.195228 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.195068    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:17.695047326 +0000 UTC m=+3.076864108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:17.200246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.200168    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7b4\" (UniqueName: \"kubernetes.io/projected/58aa49f7-f574-4eca-a269-49754a518cc1-kube-api-access-8h7b4\") pod \"aws-ebs-csi-driver-node-4vz4l\" (UID: \"58aa49f7-f574-4eca-a269-49754a518cc1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.200384 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.200348    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xvx5\" (UniqueName: \"kubernetes.io/projected/0cca3ed8-8427-4ca7-a1af-7541a27782ff-kube-api-access-5xvx5\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:17.203466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.203430    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7n8\" (UniqueName: \"kubernetes.io/projected/bc73d4b4-53e0-4357-9cc9-1a7c0697629d-kube-api-access-zw7n8\") pod \"multus-additional-cni-plugins-2gqn8\" (UID: \"bc73d4b4-53e0-4357-9cc9-1a7c0697629d\") " pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.273740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.273696    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-m5g2d"
Jul 09 16:04:17.281528 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.281488    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b78r9"
Jul 09 16:04:17.283893 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.283839    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc1043f_8908_4897_aaa0_a94fc42d5ef5.slice/crio-862b6342f12411b6f8f917174ee5809318d1f6f08bd43daa81c25836fb4f46c4 WatchSource:0}: Error finding container 862b6342f12411b6f8f917174ee5809318d1f6f08bd43daa81c25836fb4f46c4: Status 404 returned error can't find the container with id 862b6342f12411b6f8f917174ee5809318d1f6f08bd43daa81c25836fb4f46c4
Jul 09 16:04:17.289935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.289743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xsrbm"
Jul 09 16:04:17.292770 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.292741    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b54e70_0b93_411a_8cbb_6e4f68209e6a.slice/crio-4f7f42224d4a0ea2a42dc3ac6f78fe2a02dc3090590f342ea3d109076863122f WatchSource:0}: Error finding container 4f7f42224d4a0ea2a42dc3ac6f78fe2a02dc3090590f342ea3d109076863122f: Status 404 returned error can't find the container with id 4f7f42224d4a0ea2a42dc3ac6f78fe2a02dc3090590f342ea3d109076863122f
Jul 09 16:04:17.294998 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.294872    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-stdmx"
Jul 09 16:04:17.302548 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.302510    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fe573a_c004_4886_add9_ab05d9a0e8e7.slice/crio-e008e6e5b82860f19d6e1bfe73944211e561df1899211fd1011ee6ee84da26c2 WatchSource:0}: Error finding container e008e6e5b82860f19d6e1bfe73944211e561df1899211fd1011ee6ee84da26c2: Status 404 returned error can't find the container with id e008e6e5b82860f19d6e1bfe73944211e561df1899211fd1011ee6ee84da26c2
Jul 09 16:04:17.303579 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.303258    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-44htx"
Jul 09 16:04:17.312307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.312276    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:17.318689 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.318652    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8a93ba_1327_414e_9755_4c2d0ab8a9d8.slice/crio-fc94c60bec1cfba9a0de3168315b09273f6c469ad000c3d7a53c6e64fbaf3d52 WatchSource:0}: Error finding container fc94c60bec1cfba9a0de3168315b09273f6c469ad000c3d7a53c6e64fbaf3d52: Status 404 returned error can't find the container with id fc94c60bec1cfba9a0de3168315b09273f6c469ad000c3d7a53c6e64fbaf3d52
Jul 09 16:04:17.318821 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.318766    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:17.325864 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.325759    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l"
Jul 09 16:04:17.330549 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.330522    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda539fd7d_b1a9_4708_a562_1dfe76c9dc0f.slice/crio-f160a8de092dcfb01ba352988f76889707509ddba46112eff651fb7668baa3f4 WatchSource:0}: Error finding container f160a8de092dcfb01ba352988f76889707509ddba46112eff651fb7668baa3f4: Status 404 returned error can't find the container with id f160a8de092dcfb01ba352988f76889707509ddba46112eff651fb7668baa3f4
Jul 09 16:04:17.337354 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.337321    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58aa49f7_f574_4eca_a269_49754a518cc1.slice/crio-ca76769e436cdd325314987e705debd9dd5631a34a33a03b24479465b87210b8 WatchSource:0}: Error finding container ca76769e436cdd325314987e705debd9dd5631a34a33a03b24479465b87210b8: Status 404 returned error can't find the container with id ca76769e436cdd325314987e705debd9dd5631a34a33a03b24479465b87210b8
Jul 09 16:04:17.338339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.338304    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2gqn8"
Jul 09 16:04:17.350553 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:17.350515    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc73d4b4_53e0_4357_9cc9_1a7c0697629d.slice/crio-f6789808fbe52b8eb622b003ad6eebebb9bfcbe0110e5a29b8f802c0124acc65 WatchSource:0}: Error finding container f6789808fbe52b8eb622b003ad6eebebb9bfcbe0110e5a29b8f802c0124acc65: Status 404 returned error can't find the container with id f6789808fbe52b8eb622b003ad6eebebb9bfcbe0110e5a29b8f802c0124acc65
Jul 09 16:04:17.688139 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.687809    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:17.688139 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.688121    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:17.688410 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.687993    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:17.688410 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.688224    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:04:18.688199732 +0000 UTC m=+4.070016511 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:17.688410 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.688287    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:17.688410 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.688355    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:18.688338227 +0000 UTC m=+4.070154995 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:17.789284 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:17.789203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:17.789454 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.789409    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:17.789454 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.789436    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:17.789454 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.789450    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:17.789692 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:17.789519    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:18.789499842 +0000 UTC m=+4.171316626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:18.031854 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.031731    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 15:59:17 +0000 UTC" deadline="2027-12-04 23:24:52.280930505 +0000 UTC"
Jul 09 16:04:18.031854 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.031799    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12319h20m34.249134913s"
Jul 09 16:04:18.181816 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.181767    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerStarted","Data":"f6789808fbe52b8eb622b003ad6eebebb9bfcbe0110e5a29b8f802c0124acc65"}
Jul 09 16:04:18.184169 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.184062    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"551ae35e82c0b71602c28eda8f9d092fa70e93081134d104035eeafc767aa457"}
Jul 09 16:04:18.187043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.186928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44htx" event={"ID":"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8","Type":"ContainerStarted","Data":"fc94c60bec1cfba9a0de3168315b09273f6c469ad000c3d7a53c6e64fbaf3d52"}
Jul 09 16:04:18.189030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.188945    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-stdmx" event={"ID":"04b0d083-c5d4-4cc0-906d-c373319e1c97","Type":"ContainerStarted","Data":"6ced423134c40f3cd87d66f8d24b7dfad2ea1cf75114d454db9f4131a026cb8a"}
Jul 09 16:04:18.191187 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.191104    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m5g2d" event={"ID":"6dc1043f-8908-4897-aaa0-a94fc42d5ef5","Type":"ContainerStarted","Data":"862b6342f12411b6f8f917174ee5809318d1f6f08bd43daa81c25836fb4f46c4"}
Jul 09 16:04:18.192851 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.192822    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l" event={"ID":"58aa49f7-f574-4eca-a269-49754a518cc1","Type":"ContainerStarted","Data":"ca76769e436cdd325314987e705debd9dd5631a34a33a03b24479465b87210b8"}
Jul 09 16:04:18.194530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.194464    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hhw4n" event={"ID":"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f","Type":"ContainerStarted","Data":"f160a8de092dcfb01ba352988f76889707509ddba46112eff651fb7668baa3f4"}
Jul 09 16:04:18.195922 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.195829    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xsrbm" event={"ID":"06fe573a-c004-4886-add9-ab05d9a0e8e7","Type":"ContainerStarted","Data":"e008e6e5b82860f19d6e1bfe73944211e561df1899211fd1011ee6ee84da26c2"}
Jul 09 16:04:18.197155 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.197095    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b78r9" event={"ID":"41b54e70-0b93-411a-8cbb-6e4f68209e6a","Type":"ContainerStarted","Data":"4f7f42224d4a0ea2a42dc3ac6f78fe2a02dc3090590f342ea3d109076863122f"}
Jul 09 16:04:18.695410 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.695355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:18.695626 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.695462    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:18.695626 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.695609    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:18.695753 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.695682    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:04:20.695659661 +0000 UTC m=+6.077476519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:18.696077 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.695977    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:18.696077 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.696051    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:20.696031576 +0000 UTC m=+6.077848350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:18.796677 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:18.796633    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:18.796897 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.796839    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:18.796897 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.796866    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:18.796897 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.796897    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:18.797082 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:18.796960    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:20.796941129 +0000 UTC m=+6.178757900 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:19.034497 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:19.034421    2558 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 15:59:17 +0000 UTC" deadline="2028-01-03 19:26:06.725642546 +0000 UTC"
Jul 09 16:04:19.034497 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:19.034471    2558 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13035h21m47.691182807s"
Jul 09 16:04:19.172231 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:19.172193    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:19.172423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:19.172241    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:19.172423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:19.172275    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:19.172423 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:19.172394    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:19.172603 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:19.172513    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:19.172660 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:19.172601    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:19.208099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:19.207354    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal" event={"ID":"0964284bd0ac4fa0d88f69cd97267984","Type":"ContainerStarted","Data":"c1564bc438ba5909a9b2a0c543575581d4177692d712efadaba962a155f5fdfa"}
Jul 09 16:04:20.214619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:20.214577    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2a43db58e6bc067f1a769598e52ae5e" containerID="aaf0728430032d924c16a48a354915b6620e6936646e1c809ca3e4c83b993cb7" exitCode=0
Jul 09 16:04:20.215108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:20.214981    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal" event={"ID":"e2a43db58e6bc067f1a769598e52ae5e","Type":"ContainerDied","Data":"aaf0728430032d924c16a48a354915b6620e6936646e1c809ca3e4c83b993cb7"}
Jul 09 16:04:20.233715 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:20.233643    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-91.ec2.internal" podStartSLOduration=4.233618812 podStartE2EDuration="4.233618812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:19.22454915 +0000 UTC m=+4.606365939" watchObservedRunningTime="2026-07-09 16:04:20.233618812 +0000 UTC m=+5.615435601"
Jul 09 16:04:20.715470 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:20.714475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:20.715470 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:20.714636    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:20.715470 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.714814    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:20.715470 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.714901    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:04:24.714864216 +0000 UTC m=+10.096680986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:20.715470 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.715319    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:20.715470 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.715374    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:24.715357358 +0000 UTC m=+10.097174148 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:20.816836 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:20.815986    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:20.816836 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.816273    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:20.816836 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.816300    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:20.816836 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.816338    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:20.816836 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:20.816403    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:24.816381674 +0000 UTC m=+10.198198464 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:21.174718 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:21.174635    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:21.174869 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:21.174771    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:21.175221 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:21.175202    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:21.175330 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:21.175310    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:21.175392 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:21.175380    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:21.175488 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:21.175469    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:23.173606 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:23.173014    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:23.173606 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:23.173028    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:23.173606 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:23.173156    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:23.173606 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:23.173167    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:23.173606 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:23.173297    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:23.173606 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:23.173389    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:24.748642 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:24.747780    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:24.748642 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:24.747894    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:24.748642 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.748084    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:24.748642 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.748152    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:32.748133036 +0000 UTC m=+18.129949805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:24.749392 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.748688    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:24.749392 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.748759    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:04:32.748738832 +0000 UTC m=+18.130555619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:24.849080 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:24.849040    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:24.849265 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.849195    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:24.849265 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.849211    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:24.849265 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.849221    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:24.849419 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:24.849276    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:32.849257945 +0000 UTC m=+18.231074713 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:25.174145 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:25.173281    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:25.174145 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:25.173393    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:25.174145 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:25.173768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:25.174145 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:25.173898    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:25.174145 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:25.173960    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:25.174145 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:25.174051    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:27.172431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:27.172389    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:27.172431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:27.172433    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:27.173050 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:27.172389    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:27.173050 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:27.172551    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:27.173050 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:27.172600    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:27.173050 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:27.172672    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:29.172897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.172695    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:29.173348 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.172709    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:29.173348 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:29.173009    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:29.173348 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.172729    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:29.173348 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:29.173101    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:29.181473 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:29.181372    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:29.236529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.236487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-44htx" event={"ID":"bd8a93ba-1327-414e-9755-4c2d0ab8a9d8","Type":"ContainerStarted","Data":"2ddab069a21315202965a8b3cb70fa68ae0ddc6c336b00fbae5a1003eddd7b90"}
Jul 09 16:04:29.238270 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.238230    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-stdmx" event={"ID":"04b0d083-c5d4-4cc0-906d-c373319e1c97","Type":"ContainerStarted","Data":"e782c739751467941146840736599d8413edbb63a143b627e569e580d0a4c45d"}
Jul 09 16:04:29.240663 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.240602    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal" event={"ID":"e2a43db58e6bc067f1a769598e52ae5e","Type":"ContainerStarted","Data":"edef39f9400cd55f61337fc0615010a47ca2550859f8c2a42a4bfecb0e1cc473"}
Jul 09 16:04:29.242529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.242490    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l" event={"ID":"58aa49f7-f574-4eca-a269-49754a518cc1","Type":"ContainerStarted","Data":"619662e9b071b92e09d41fc2ba471adebee9b7843fe6fd9ebe4b2a212fa83e29"}
Jul 09 16:04:29.244394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.244355    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hhw4n" event={"ID":"a539fd7d-b1a9-4708-a562-1dfe76c9dc0f","Type":"ContainerStarted","Data":"eaf29dc3e2377dd898045e8a3f593960e6acc8588c7aacb67bfb41dc778cdd81"}
Jul 09 16:04:29.246379 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.246347    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xsrbm" event={"ID":"06fe573a-c004-4886-add9-ab05d9a0e8e7","Type":"ContainerStarted","Data":"e7677207e542b416de0d05900d2487a859196f4ba0fa33a4e72324f55d34551e"}
Jul 09 16:04:29.248319 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.248284    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc73d4b4-53e0-4357-9cc9-1a7c0697629d" containerID="b01d36a7d1a2740b130532d015ea1dc07e0a314e5cd33f6369f3b6c619cb859e" exitCode=0
Jul 09 16:04:29.248445 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.248336    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerDied","Data":"b01d36a7d1a2740b130532d015ea1dc07e0a314e5cd33f6369f3b6c619cb859e"}
Jul 09 16:04:29.254500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.254432    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-44htx" podStartSLOduration=3.044959367 podStartE2EDuration="14.254408001s" totalImagesPullingTime="11.209448634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:29.253926769 +0000 UTC m=+14.635743548" watchObservedRunningTime="2026-07-09 16:04:29.254408001 +0000 UTC m=+14.636224791"
Jul 09 16:04:29.270474 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.270202    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-91.ec2.internal" podStartSLOduration=13.270183215 podStartE2EDuration="13.270183215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:29.269579648 +0000 UTC m=+14.651396437" watchObservedRunningTime="2026-07-09 16:04:29.270183215 +0000 UTC m=+14.652000002"
Jul 09 16:04:29.290355 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.290302    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xsrbm" podStartSLOduration=2.739588753 podStartE2EDuration="14.290278586s" totalImagesPullingTime="11.550689833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:29.289590976 +0000 UTC m=+14.671407764" watchObservedRunningTime="2026-07-09 16:04:29.290278586 +0000 UTC m=+14.672095394"
Jul 09 16:04:29.327456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.327406    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-stdmx" podStartSLOduration=3.107309295 podStartE2EDuration="14.327385586s" totalImagesPullingTime="11.220076291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:29.327227753 +0000 UTC m=+14.709044540" watchObservedRunningTime="2026-07-09 16:04:29.327385586 +0000 UTC m=+14.709202370"
Jul 09 16:04:29.344804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.344735    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hhw4n" podStartSLOduration=3.147747161 podStartE2EDuration="14.344717795s" totalImagesPullingTime="11.196970634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:29.344152554 +0000 UTC m=+14.725969342" watchObservedRunningTime="2026-07-09 16:04:29.344717795 +0000 UTC m=+14.726534582"
Jul 09 16:04:29.839191 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.839158    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:29.839954 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:29.839920    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:30.252154 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:30.252062    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b78r9" event={"ID":"41b54e70-0b93-411a-8cbb-6e4f68209e6a","Type":"ContainerStarted","Data":"9ea086f2f8507423a83cd26f8644cf299f9089c4c30bfa5b919b468148be1178"}
Jul 09 16:04:30.268064 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:30.267993    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b78r9" podStartSLOduration=4.030924661 podStartE2EDuration="15.267970085s" totalImagesPullingTime="11.237045424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:30.267827859 +0000 UTC m=+15.649644647" watchObservedRunningTime="2026-07-09 16:04:30.267970085 +0000 UTC m=+15.649786874"
Jul 09 16:04:31.172298 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:31.172249    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:31.172492 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:31.172400    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:31.172492 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:31.172249    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:31.172621 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:31.172566    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:31.172677 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:31.172635    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:31.172726 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:31.172706    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:31.254501 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:31.254466    2558 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 16:04:31.477205 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:31.477113    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:31.477930 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:31.477906    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hhw4n"
Jul 09 16:04:32.815935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:32.815889    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:32.816689 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:32.815985    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:32.816689 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.816052    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:32.816689 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.816074    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:32.816689 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.816142    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:04:48.816120374 +0000 UTC m=+34.197937140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:32.816689 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.816165    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:48.816154786 +0000 UTC m=+34.197971554 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:32.917054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:32.917013    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:32.917320 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.917227    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:32.917320 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.917249    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:32.917320 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.917261    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:32.917472 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:32.917332    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:04:48.917312414 +0000 UTC m=+34.299129200 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:33.173201 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:33.173104    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:33.173201 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:33.173126    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:33.173201 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:33.173126    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:33.173457 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:33.173246    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:33.173457 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:33.173372    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:33.173542 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:33.173462    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:35.174900 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:35.174625    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:35.175423 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:35.175019    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:35.175423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:35.175085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:35.175423 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:35.175206    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:35.175423 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:35.175355    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:35.175641 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:35.175463    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:37.172400 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:37.172363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:37.172966 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:37.172363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:37.172966 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:37.172497    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:37.172966 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:37.172560    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:37.172966 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:37.172363    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:37.172966 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:37.172634    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:37.848547 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:37.848522    2558 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 16:04:38.064339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.064175    2558 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T16:04:37.848543313Z","UUID":"ea65c14a-1100-4556-bedc-625e8bc95161","Handler":null,"Name":"","Endpoint":""}
Jul 09 16:04:38.065595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.065575    2558 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 16:04:38.065688 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.065617    2558 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 16:04:38.269925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.269894    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l" event={"ID":"58aa49f7-f574-4eca-a269-49754a518cc1","Type":"ContainerStarted","Data":"468f65bf3c60beb3d4c5188d48f74c07026d0eaef5e228260d6ed98a7a50b859"}
Jul 09 16:04:38.272834 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.272814    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:04:38.273198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273177    2558 generic.go:356] "Generic (PLEG): container finished" podID="d30497fb-b5e7-45da-af81-0ff51251b144" containerID="b612d3113f16f8aec155214b3b4955a47109449e2fb483bfeb426df78f2d9012" exitCode=1
Jul 09 16:04:38.273254 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273207    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"ce942ce2060b3a3ed4320223da75cdf7e06573fdfaf1dd67fc6cd03e3dfe496f"}
Jul 09 16:04:38.273307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273283    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"735b4b11e19b9ddc650d8dfc69cc045e7a23cf9c65560d24f9f97b753909f6f4"}
Jul 09 16:04:38.273367 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"6154a182ad049ef84780eba630c85929b3b04a85b9d177b49c3ca1396b3a85ae"}
Jul 09 16:04:38.273419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273367    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"17a00ce2597f59bababa171223e1ee632332bce7d6afea2ee0d151ca79a82fb0"}
Jul 09 16:04:38.273419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273407    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerDied","Data":"b612d3113f16f8aec155214b3b4955a47109449e2fb483bfeb426df78f2d9012"}
Jul 09 16:04:38.273497 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.273446    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"bf97ad308f13d471375fb0f5b352e4ca27533b95370d10e9fa7981dd35aa73e3"}
Jul 09 16:04:38.274810 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.274785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-m5g2d" event={"ID":"6dc1043f-8908-4897-aaa0-a94fc42d5ef5","Type":"ContainerStarted","Data":"df4f46b45dec0b2e8aa5a7c42ec85a8ec659866d98f92e09578c6edfe81ebb32"}
Jul 09 16:04:38.292538 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:38.292487    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-m5g2d" podStartSLOduration=2.873835429 podStartE2EDuration="23.292469987s" totalImagesPullingTime="20.418634558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:38.292385696 +0000 UTC m=+23.674202485" watchObservedRunningTime="2026-07-09 16:04:38.292469987 +0000 UTC m=+23.674286776"
Jul 09 16:04:39.172721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:39.172693    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:39.172952 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:39.172836    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:39.173070 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:39.173047    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:39.173190 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:39.173161    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:39.173258 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:39.173159    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:39.173319 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:39.173258    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:39.279295 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:39.279265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l" event={"ID":"58aa49f7-f574-4eca-a269-49754a518cc1","Type":"ContainerStarted","Data":"08f58d4d510f7255a521a487f06d12bbbb8480834fa25c3b1db77aecd34926fb"}
Jul 09 16:04:39.298121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:39.298057    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4vz4l" podStartSLOduration=3.031922394 podStartE2EDuration="24.298038326s" totalImagesPullingTime="21.266115932s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:39.297581193 +0000 UTC m=+24.679397985" watchObservedRunningTime="2026-07-09 16:04:39.298038326 +0000 UTC m=+24.679855114"
Jul 09 16:04:41.172666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:41.172627    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:41.172666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:41.172658    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:41.173123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:41.172630    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:41.173123 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:41.172744    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:41.173123 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:41.172866    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:41.173123 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:41.172993    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:41.286130 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:41.286104    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:04:41.286450 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:41.286428    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"a0bc3e1e4436290940d6ce9c28bab90af191c7d0ee1a53d9f9aa5c95a5ce67fd"}
Jul 09 16:04:43.172317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:43.172279    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:43.172870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:43.172280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:43.172870 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:43.172423    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:43.172870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:43.172280    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:43.172870 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:43.172511    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:43.172870 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:43.172572    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:44.297490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.297317    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc73d4b4-53e0-4357-9cc9-1a7c0697629d" containerID="62a99a1593df562c25dc0ad7401e7e33d54db94a6251cfa4d3cee8eb9e7395ef" exitCode=0
Jul 09 16:04:44.298117 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.297402    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerDied","Data":"62a99a1593df562c25dc0ad7401e7e33d54db94a6251cfa4d3cee8eb9e7395ef"}
Jul 09 16:04:44.300912 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.300894    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:04:44.301236 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.301218    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"56e1de26fc3c0b88aad616d57a573456893e808b12d19c13843b31403756dec0"}
Jul 09 16:04:44.301475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.301456    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:44.301547 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.301526    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:44.301679 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.301665    2558 scope.go:117] "RemoveContainer" containerID="b612d3113f16f8aec155214b3b4955a47109449e2fb483bfeb426df78f2d9012"
Jul 09 16:04:44.317718 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:44.317698    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:45.173781 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.173753    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:45.173924 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:45.173867    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:45.174340 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.174302    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:45.174434 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:45.174414    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:45.174499 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.174472    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:45.174612 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:45.174539    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:45.306540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.306515    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:04:45.306968 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.306832    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" event={"ID":"d30497fb-b5e7-45da-af81-0ff51251b144","Type":"ContainerStarted","Data":"f5863fe76ef0eef8ae86f9e8bd82e78eab655115bed272b86b959c98bbca8dc7"}
Jul 09 16:04:45.307017 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.307005    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:45.325069 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.325029    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:04:45.337241 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.337199    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njhlg"]
Jul 09 16:04:45.337408 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.337336    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:45.337466 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:45.337447    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:45.337849 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.337826    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9k9w"]
Jul 09 16:04:45.338004 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.337945    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:45.338153 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:45.338130    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:45.338437 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.338418    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wzgz"]
Jul 09 16:04:45.338539 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.338526    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:45.338622 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:45.338605    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:45.343613 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:45.343573    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt" podStartSLOduration=9.914626907 podStartE2EDuration="30.343557725s" totalImagesPullingTime="20.428930818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:45.343216855 +0000 UTC m=+30.725033656" watchObservedRunningTime="2026-07-09 16:04:45.343557725 +0000 UTC m=+30.725374512"
Jul 09 16:04:46.311316 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:46.311137    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc73d4b4-53e0-4357-9cc9-1a7c0697629d" containerID="03b5acc2e7bc9039e9d1651efa783d9345424d3d5c95ffb33b14dab2e945adb8" exitCode=0
Jul 09 16:04:46.311719 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:46.311223    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerDied","Data":"03b5acc2e7bc9039e9d1651efa783d9345424d3d5c95ffb33b14dab2e945adb8"}
Jul 09 16:04:47.172342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:47.172308    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:47.172545 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:47.172309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:47.172545 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:47.172415    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:47.172644 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:47.172545    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:47.172644 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:47.172576    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:47.172726 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:47.172670    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:48.839145 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:48.839123    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:48.839600 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:48.839159    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:48.839600 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.839268    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:48.839600 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.839320    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:05:20.839307112 +0000 UTC m=+66.221123879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:04:48.839600 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.839268    2558 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:48.839600 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.839382    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret podName:ed33335f-5aec-4360-9937-e478cd9fbe04 nodeName:}" failed. No retries permitted until 2026-07-09 16:05:20.839371115 +0000 UTC m=+66.221187883 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret") pod "global-pull-secret-syncer-x9k9w" (UID: "ed33335f-5aec-4360-9937-e478cd9fbe04") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:04:48.940066 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:48.940002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:48.940233 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.940161    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:04:48.940233 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.940183    2558 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:04:48.940233 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.940193    2558 projected.go:196] Error preparing data for projected volume kube-api-access-j6tcd for pod openshift-network-diagnostics/network-check-target-njhlg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:48.940339 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:48.940242    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd podName:7840aa1c-6746-4c6b-8b0a-ad804712c056 nodeName:}" failed. No retries permitted until 2026-07-09 16:05:20.940228267 +0000 UTC m=+66.322045037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-j6tcd" (UniqueName: "kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd") pod "network-check-target-njhlg" (UID: "7840aa1c-6746-4c6b-8b0a-ad804712c056") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:04:49.171896 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:49.171858    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:49.172087 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:49.171866    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:49.172087 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:49.172005    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-njhlg" podUID="7840aa1c-6746-4c6b-8b0a-ad804712c056"
Jul 09 16:04:49.172087 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:49.171869    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:49.172087 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:49.172067    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:04:49.172283 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:49.172137    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x9k9w" podUID="ed33335f-5aec-4360-9937-e478cd9fbe04"
Jul 09 16:04:49.319668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:49.319636    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc73d4b4-53e0-4357-9cc9-1a7c0697629d" containerID="e402d45fc38309e8ad76e1753f65afa6536f1e69bb635c4054b675258503e903" exitCode=0
Jul 09 16:04:49.319854 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:49.319694    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerDied","Data":"e402d45fc38309e8ad76e1753f65afa6536f1e69bb635c4054b675258503e903"}
Jul 09 16:04:50.426449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.426375    2558 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-133-91.ec2.internal" event="NodeReady"
Jul 09 16:04:50.426845 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.426507    2558 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 16:04:50.474073 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.474023    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fhktg"]
Jul 09 16:04:50.476142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.476127    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jj4z8"]
Jul 09 16:04:50.476272 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.476256    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.477936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.477920    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.479363 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.479343    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvt5b\""
Jul 09 16:04:50.479454 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.479379    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 16:04:50.479454 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.479414    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 16:04:50.480683 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.480666    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 16:04:50.480771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.480670    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 16:04:50.480978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.480961    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 16:04:50.481015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.480992    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-ddkpg\""
Jul 09 16:04:50.488604 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.488586    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jj4z8"]
Jul 09 16:04:50.488712 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.488625    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fhktg"]
Jul 09 16:04:50.553385 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.553356    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sg74\" (UniqueName: \"kubernetes.io/projected/36639cf2-b21a-4d83-b9ba-e07f0412759d-kube-api-access-7sg74\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.553518 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.553405    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7add5d7-9318-4518-ac96-eb8d62dfc423-cert\") pod \"ingress-canary-jj4z8\" (UID: \"a7add5d7-9318-4518-ac96-eb8d62dfc423\") " pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.553518 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.553455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srss2\" (UniqueName: \"kubernetes.io/projected/a7add5d7-9318-4518-ac96-eb8d62dfc423-kube-api-access-srss2\") pod \"ingress-canary-jj4z8\" (UID: \"a7add5d7-9318-4518-ac96-eb8d62dfc423\") " pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.553518 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.553483    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36639cf2-b21a-4d83-b9ba-e07f0412759d-tmp-dir\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.553623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.553521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.553623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.553549    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36639cf2-b21a-4d83-b9ba-e07f0412759d-config-volume\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.654104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654075    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7add5d7-9318-4518-ac96-eb8d62dfc423-cert\") pod \"ingress-canary-jj4z8\" (UID: \"a7add5d7-9318-4518-ac96-eb8d62dfc423\") " pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.654260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654123    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srss2\" (UniqueName: \"kubernetes.io/projected/a7add5d7-9318-4518-ac96-eb8d62dfc423-kube-api-access-srss2\") pod \"ingress-canary-jj4z8\" (UID: \"a7add5d7-9318-4518-ac96-eb8d62dfc423\") " pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.654260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36639cf2-b21a-4d83-b9ba-e07f0412759d-tmp-dir\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.654260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654182    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.654260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654206    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36639cf2-b21a-4d83-b9ba-e07f0412759d-config-volume\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.654260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sg74\" (UniqueName: \"kubernetes.io/projected/36639cf2-b21a-4d83-b9ba-e07f0412759d-kube-api-access-7sg74\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.654491 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:50.654459    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:04:50.654530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654490    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36639cf2-b21a-4d83-b9ba-e07f0412759d-tmp-dir\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.654530 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:50.654513    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:04:51.154499283 +0000 UTC m=+36.536316053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:04:50.654857 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.654831    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36639cf2-b21a-4d83-b9ba-e07f0412759d-config-volume\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.659178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.659159    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a7add5d7-9318-4518-ac96-eb8d62dfc423-cert\") pod \"ingress-canary-jj4z8\" (UID: \"a7add5d7-9318-4518-ac96-eb8d62dfc423\") " pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.664046 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.664023    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srss2\" (UniqueName: \"kubernetes.io/projected/a7add5d7-9318-4518-ac96-eb8d62dfc423-kube-api-access-srss2\") pod \"ingress-canary-jj4z8\" (UID: \"a7add5d7-9318-4518-ac96-eb8d62dfc423\") " pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.664252 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.664231    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sg74\" (UniqueName: \"kubernetes.io/projected/36639cf2-b21a-4d83-b9ba-e07f0412759d-kube-api-access-7sg74\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:50.792559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.792480    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jj4z8"
Jul 09 16:04:50.974253 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:50.974028    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jj4z8"]
Jul 09 16:04:50.976944 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:50.976916    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7add5d7_9318_4518_ac96_eb8d62dfc423.slice/crio-149bce798686ef1ee06b342414cd751ff46591b07d12f548060cd93389b17617 WatchSource:0}: Error finding container 149bce798686ef1ee06b342414cd751ff46591b07d12f548060cd93389b17617: Status 404 returned error can't find the container with id 149bce798686ef1ee06b342414cd751ff46591b07d12f548060cd93389b17617
Jul 09 16:04:51.158275 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.158241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:51.158465 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:51.158371    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:04:51.158465 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:51.158436    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:04:52.158417675 +0000 UTC m=+37.540234441 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:04:51.171890 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.171854    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:04:51.171890 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.171864    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:04:51.172151 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.172136    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:04:51.175083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.175063    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 16:04:51.175194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.175089    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 16:04:51.175194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.175182    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 16:04:51.175194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.175190    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vzqd2\""
Jul 09 16:04:51.175352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.175063    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 16:04:51.175543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.175482    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdktf\""
Jul 09 16:04:51.325583 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:51.325555    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jj4z8" event={"ID":"a7add5d7-9318-4518-ac96-eb8d62dfc423","Type":"ContainerStarted","Data":"149bce798686ef1ee06b342414cd751ff46591b07d12f548060cd93389b17617"}
Jul 09 16:04:52.167692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:52.167654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:52.168322 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:52.167831    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:04:52.168322 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:52.167936    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:04:54.167912463 +0000 UTC m=+39.549729244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:04:53.553671 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.553638    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"]
Jul 09 16:04:53.575181 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.575155    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"]
Jul 09 16:04:53.575346 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.575284    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.577707 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.577681    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"]
Jul 09 16:04:53.578629 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.578590    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 16:04:53.578629 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.578596    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 16:04:53.578799 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.578596    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 16:04:53.578799 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.578747    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 16:04:53.599284 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.599251    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"]
Jul 09 16:04:53.599393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.599367    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.602530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.602507    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 16:04:53.602530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.602521    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 16:04:53.602687 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.602605    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 16:04:53.602687 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.602682    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 16:04:53.680804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.680770    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.680804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.680807    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-ca\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.681037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.680922    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.681037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.680963    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmlr\" (UniqueName: \"kubernetes.io/projected/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-kube-api-access-wqmlr\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.681037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.680984    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-tmp\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.681194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.681040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.681194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.681056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdhs6\" (UniqueName: \"kubernetes.io/projected/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-kube-api-access-pdhs6\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.681194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.681110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-hub\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.681194 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.681144    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-klusterlet-config\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.781759 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.781717    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.782454 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782422    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmlr\" (UniqueName: \"kubernetes.io/projected/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-kube-api-access-wqmlr\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.782593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782466    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-tmp\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.782593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782505    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.782593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdhs6\" (UniqueName: \"kubernetes.io/projected/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-kube-api-access-pdhs6\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.782593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782553    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-hub\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.782593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-klusterlet-config\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.782805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.782805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.782639    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-ca\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.783250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.783218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-tmp\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.784207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.783862    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.786332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.786303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-ca\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.786655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.786542    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.786655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.786598    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-klusterlet-config\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.786655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.786628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-hub\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.786804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.786776    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.792952 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.792930    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmlr\" (UniqueName: \"kubernetes.io/projected/45e82465-f8ac-4c6e-ba5f-b201f5b0c118-kube-api-access-wqmlr\") pod \"cluster-proxy-proxy-agent-6b4d969697-z6vv7\" (UID: \"45e82465-f8ac-4c6e-ba5f-b201f5b0c118\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:53.793347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.793328    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdhs6\" (UniqueName: \"kubernetes.io/projected/bd7d27ed-ffd8-4378-bae0-80e4c0c0e343-kube-api-access-pdhs6\") pod \"klusterlet-addon-workmgr-894754c84-l2kgq\" (UID: \"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.886484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.886444    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:04:53.922337 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:53.922301    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:04:54.185737 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:54.185644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:54.185920 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:54.185821    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:04:54.185992 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:54.185928    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:04:58.185906497 +0000 UTC m=+43.567723278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:04:56.514735 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:56.514572    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"]
Jul 09 16:04:56.517824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:56.517801    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"]
Jul 09 16:04:56.518621 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:56.518598    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45e82465_f8ac_4c6e_ba5f_b201f5b0c118.slice/crio-fbf9e8148e6466be3d78377bd7c7a29ab0a531161b1ffd967ea3af0b42137d52 WatchSource:0}: Error finding container fbf9e8148e6466be3d78377bd7c7a29ab0a531161b1ffd967ea3af0b42137d52: Status 404 returned error can't find the container with id fbf9e8148e6466be3d78377bd7c7a29ab0a531161b1ffd967ea3af0b42137d52
Jul 09 16:04:56.522318 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:04:56.522293    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7d27ed_ffd8_4378_bae0_80e4c0c0e343.slice/crio-9eb29076d6580f7f9db4e3f462ab9954480559e1a00c8edf9272ab4bdf4beb98 WatchSource:0}: Error finding container 9eb29076d6580f7f9db4e3f462ab9954480559e1a00c8edf9272ab4bdf4beb98: Status 404 returned error can't find the container with id 9eb29076d6580f7f9db4e3f462ab9954480559e1a00c8edf9272ab4bdf4beb98
Jul 09 16:04:57.348837 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:57.348802    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc73d4b4-53e0-4357-9cc9-1a7c0697629d" containerID="c8a435f1c45639b8a76989e5501b676ed2b583365b6c3835d4157c64264835d6" exitCode=0
Jul 09 16:04:57.349007 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:57.348925    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerDied","Data":"c8a435f1c45639b8a76989e5501b676ed2b583365b6c3835d4157c64264835d6"}
Jul 09 16:04:57.351826 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:57.351786    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jj4z8" event={"ID":"a7add5d7-9318-4518-ac96-eb8d62dfc423","Type":"ContainerStarted","Data":"d19ab6e980bddebe2ab3a027b13a29bec38e55b04ff313a9c106cad784179ef7"}
Jul 09 16:04:57.355592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:57.355562    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq" event={"ID":"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343","Type":"ContainerStarted","Data":"9eb29076d6580f7f9db4e3f462ab9954480559e1a00c8edf9272ab4bdf4beb98"}
Jul 09 16:04:57.357534 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:57.357508    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" event={"ID":"45e82465-f8ac-4c6e-ba5f-b201f5b0c118","Type":"ContainerStarted","Data":"fbf9e8148e6466be3d78377bd7c7a29ab0a531161b1ffd967ea3af0b42137d52"}
Jul 09 16:04:57.391496 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:57.391454    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jj4z8" podStartSLOduration=1.996134834 podStartE2EDuration="7.391435708s" totalImagesPullingTime="5.395300874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:04:57.390526923 +0000 UTC m=+42.772343723" watchObservedRunningTime="2026-07-09 16:04:57.391435708 +0000 UTC m=+42.773252474"
Jul 09 16:04:58.221603 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:58.221562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:04:58.222411 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:58.221720    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:04:58.222411 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:04:58.221783    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:05:06.221764209 +0000 UTC m=+51.603580999 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:04:58.369199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:58.368091    2558 generic.go:356] "Generic (PLEG): container finished" podID="bc73d4b4-53e0-4357-9cc9-1a7c0697629d" containerID="e2c0abf9c5e3f3494f71916c83d1d6a68d19dfe7f7f217a22d3269e20ceb27cd" exitCode=0
Jul 09 16:04:58.369199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:04:58.369026    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerDied","Data":"e2c0abf9c5e3f3494f71916c83d1d6a68d19dfe7f7f217a22d3269e20ceb27cd"}
Jul 09 16:05:00.377999 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:00.377781    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" event={"ID":"bc73d4b4-53e0-4357-9cc9-1a7c0697629d","Type":"ContainerStarted","Data":"957041a960831e280815d6dca8c91855d5ded6bf0689ba41b4f6bf0690ce036c"}
Jul 09 16:05:00.403202 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:00.403137    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2gqn8" podStartSLOduration=9.213632847 podStartE2EDuration="45.403115797s" totalImagesPullingTime="36.18948295s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:00.401387716 +0000 UTC m=+45.783204504" watchObservedRunningTime="2026-07-09 16:05:00.403115797 +0000 UTC m=+45.784932585"
Jul 09 16:05:03.387947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.387735    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" event={"ID":"45e82465-f8ac-4c6e-ba5f-b201f5b0c118","Type":"ContainerStarted","Data":"2de1ff7d05ea3c36834950a886f3d0182c32bb1d2467b4e8b120474ac9d1458f"}
Jul 09 16:05:03.388405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.387973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" event={"ID":"45e82465-f8ac-4c6e-ba5f-b201f5b0c118","Type":"ContainerStarted","Data":"548cd3eb3368b3dc1705546746766b7238fe33f830616318c3e98714a96c0f60"}
Jul 09 16:05:03.388405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.387999    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" event={"ID":"45e82465-f8ac-4c6e-ba5f-b201f5b0c118","Type":"ContainerStarted","Data":"c4e346d2fef6270da868819045e42ef9dc13ecfd4053f0be4e5b6f13b6e0ac23"}
Jul 09 16:05:03.392206 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.392170    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq" event={"ID":"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343","Type":"ContainerStarted","Data":"1119986fbd86d4ce58c7dc92054c301dced3f8e683aca83f83a2c1619dceaefb"}
Jul 09 16:05:03.392405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.392390    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:05:03.394136 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.394110    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:05:03.406751 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.406706    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" podStartSLOduration=4.543158097 podStartE2EDuration="10.406690021s" totalImagesPullingTime="5.863531924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:03.405812758 +0000 UTC m=+48.787629545" watchObservedRunningTime="2026-07-09 16:05:03.406690021 +0000 UTC m=+48.788506812"
Jul 09 16:05:03.426800 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:03.426740    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq" podStartSLOduration=4.553594896 podStartE2EDuration="10.42672045s" totalImagesPullingTime="5.873125554s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:03.426379769 +0000 UTC m=+48.808196558" watchObservedRunningTime="2026-07-09 16:05:03.42672045 +0000 UTC m=+48.808537239"
Jul 09 16:05:06.284862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:06.284824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:05:06.285284 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:06.284986    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:05:06.285284 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:06.285049    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:05:22.285033655 +0000 UTC m=+67.666850421 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:05:17.324552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:17.324521    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-px6rt"
Jul 09 16:05:20.885862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.885811    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:05:20.886269 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.885896    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:05:20.889202 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.889183    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 16:05:20.889272 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.889213    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 16:05:20.896737 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:20.896720    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 16:05:20.896789 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:20.896779    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:06:24.896761681 +0000 UTC m=+130.278578447 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : secret "metrics-daemon-secret" not found
Jul 09 16:05:20.900528 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.900512    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ed33335f-5aec-4360-9937-e478cd9fbe04-original-pull-secret\") pod \"global-pull-secret-syncer-x9k9w\" (UID: \"ed33335f-5aec-4360-9937-e478cd9fbe04\") " pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:05:20.987169 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.987142    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:05:20.990137 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.990123    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 16:05:20.999989 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:20.999973    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 16:05:21.010431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.010412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6tcd\" (UniqueName: \"kubernetes.io/projected/7840aa1c-6746-4c6b-8b0a-ad804712c056-kube-api-access-j6tcd\") pod \"network-check-target-njhlg\" (UID: \"7840aa1c-6746-4c6b-8b0a-ad804712c056\") " pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:05:21.183447 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.183391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x9k9w"
Jul 09 16:05:21.198416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.198396    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vzqd2\""
Jul 09 16:05:21.205787 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.205768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:05:21.310529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.310498    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x9k9w"]
Jul 09 16:05:21.313981 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:05:21.313949    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded33335f_5aec_4360_9937_e478cd9fbe04.slice/crio-55c31134b558056548767552ce918fbf01eac4badb91c36803686c90dd5ede39 WatchSource:0}: Error finding container 55c31134b558056548767552ce918fbf01eac4badb91c36803686c90dd5ede39: Status 404 returned error can't find the container with id 55c31134b558056548767552ce918fbf01eac4badb91c36803686c90dd5ede39
Jul 09 16:05:21.332564 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.332538    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-njhlg"]
Jul 09 16:05:21.334836 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:05:21.334814    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7840aa1c_6746_4c6b_8b0a_ad804712c056.slice/crio-a879e95f4a34e7b488e61bd27b64a092a7f50e313a04b4b284818332447d6005 WatchSource:0}: Error finding container a879e95f4a34e7b488e61bd27b64a092a7f50e313a04b4b284818332447d6005: Status 404 returned error can't find the container with id a879e95f4a34e7b488e61bd27b64a092a7f50e313a04b4b284818332447d6005
Jul 09 16:05:21.437369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.437285    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njhlg" event={"ID":"7840aa1c-6746-4c6b-8b0a-ad804712c056","Type":"ContainerStarted","Data":"a879e95f4a34e7b488e61bd27b64a092a7f50e313a04b4b284818332447d6005"}
Jul 09 16:05:21.438291 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:21.438267    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9k9w" event={"ID":"ed33335f-5aec-4360-9937-e478cd9fbe04","Type":"ContainerStarted","Data":"55c31134b558056548767552ce918fbf01eac4badb91c36803686c90dd5ede39"}
Jul 09 16:05:22.296439 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:22.296408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:05:22.296439 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:22.296542    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:05:22.297161 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:22.296611    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:05:54.296590934 +0000 UTC m=+99.678407713 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:05:27.457765 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:27.457724    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x9k9w" event={"ID":"ed33335f-5aec-4360-9937-e478cd9fbe04","Type":"ContainerStarted","Data":"fd7ec38da723dde5f72b0d28345d8687c39e5f14c7cc86800b7326f56503f6c1"}
Jul 09 16:05:27.459825 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:27.459752    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-njhlg" event={"ID":"7840aa1c-6746-4c6b-8b0a-ad804712c056","Type":"ContainerStarted","Data":"698e01f63d4ad182f1ceca452a163d8ddd8803697debe76ada622793fec1d3ee"}
Jul 09 16:05:27.459979 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:27.459928    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:05:27.493813 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:27.493766    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x9k9w" podStartSLOduration=66.535911718 podStartE2EDuration="1m12.493751661s" totalImagesPullingTime="5.957839943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:27.493160172 +0000 UTC m=+72.874976962" watchObservedRunningTime="2026-07-09 16:05:27.493751661 +0000 UTC m=+72.875568448"
Jul 09 16:05:54.312230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:54.312199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:05:54.312675 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:54.312441    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:05:54.312675 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:05:54.312533    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:06:58.312511292 +0000 UTC m=+163.694328081 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:05:58.465836 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:58.465799    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-njhlg"
Jul 09 16:05:58.481864 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:05:58.481816    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-njhlg" podStartSLOduration=97.5348849 podStartE2EDuration="1m43.481784596s" totalImagesPullingTime="5.946899696s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:05:27.529518184 +0000 UTC m=+72.911334972" watchObservedRunningTime="2026-07-09 16:05:58.481784596 +0000 UTC m=+103.863601365"
Jul 09 16:06:24.923054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:06:24.922998    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:06:24.923533 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:06:24.923158    2558 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 09 16:06:24.923533 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:06:24.923232    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs podName:0cca3ed8-8427-4ca7-a1af-7541a27782ff nodeName:}" failed. No retries permitted until 2026-07-09 16:08:26.923213436 +0000 UTC m=+252.305030207 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs") pod "network-metrics-daemon-6wzgz" (UID: "0cca3ed8-8427-4ca7-a1af-7541a27782ff") : secret "metrics-daemon-secret" not found
Jul 09 16:06:53.486589 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:06:53.486542    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fhktg" podUID="36639cf2-b21a-4d83-b9ba-e07f0412759d"
Jul 09 16:06:53.700699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:06:53.700668    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fhktg"
Jul 09 16:06:54.189934 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:06:54.189865    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-6wzgz" podUID="0cca3ed8-8427-4ca7-a1af-7541a27782ff"
Jul 09 16:06:58.341860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:06:58.341821    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:06:58.342271 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:06:58.341962    2558 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:06:58.342271 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:06:58.342030    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls podName:36639cf2-b21a-4d83-b9ba-e07f0412759d nodeName:}" failed. No retries permitted until 2026-07-09 16:09:00.342012839 +0000 UTC m=+285.723829604 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls") pod "dns-default-fhktg" (UID: "36639cf2-b21a-4d83-b9ba-e07f0412759d") : secret "dns-default-metrics-tls" not found
Jul 09 16:07:02.727009 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:02.726969    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq" event={"ID":"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343","Type":"ContainerDied","Data":"1119986fbd86d4ce58c7dc92054c301dced3f8e683aca83f83a2c1619dceaefb"}
Jul 09 16:07:02.727009 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:02.726990    2558 generic.go:356] "Generic (PLEG): container finished" podID="bd7d27ed-ffd8-4378-bae0-80e4c0c0e343" containerID="1119986fbd86d4ce58c7dc92054c301dced3f8e683aca83f83a2c1619dceaefb" exitCode=1
Jul 09 16:07:02.727520 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:02.727381    2558 scope.go:117] "RemoveContainer" containerID="1119986fbd86d4ce58c7dc92054c301dced3f8e683aca83f83a2c1619dceaefb"
Jul 09 16:07:03.392904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:03.392826    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:07:03.731623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:03.731531    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq" event={"ID":"bd7d27ed-ffd8-4378-bae0-80e4c0c0e343","Type":"ContainerStarted","Data":"dc6e9ac27b7d9b7553055dc08f7269e01b3dfa6cfb11a4e7a7bd72061e0795d0"}
Jul 09 16:07:03.731987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:03.731753    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:07:03.732334 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:03.732314    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-894754c84-l2kgq"
Jul 09 16:07:07.172090 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:07.172000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:07:55.781037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.780997    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2x8xz"]
Jul 09 16:07:55.783288 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.783267    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.785949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.785924    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 16:07:55.786071 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.785985    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 16:07:55.787417 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.787395    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-hnt4w\""
Jul 09 16:07:55.787532 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.787416    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 16:07:55.787532 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.787422    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 16:07:55.800381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.797263    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2x8xz"]
Jul 09 16:07:55.853233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.853198    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.853233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.853232    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69gxk\" (UniqueName: \"kubernetes.io/projected/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-api-access-69gxk\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.853412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.853322    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-data-volume\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.853412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.853355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-crio-socket\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.853412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.853395    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.954687 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.954644    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-data-volume\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.954687 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.954684    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-crio-socket\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.954951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.954704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.954951 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:55.954765    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:07:55.954951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.954790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-crio-socket\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.954951 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:55.954821    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:07:56.45480397 +0000 UTC m=+221.836620736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : configmap "kube-rbac-proxy" not found
Jul 09 16:07:55.954951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.954850    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.954951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.954870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69gxk\" (UniqueName: \"kubernetes.io/projected/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-api-access-69gxk\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.955214 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:55.954983    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:55.955214 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.955006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-data-volume\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:55.955214 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:55.955052    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:07:56.455036339 +0000 UTC m=+221.836853107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:55.965595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:55.965557    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69gxk\" (UniqueName: \"kubernetes.io/projected/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-api-access-69gxk\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:56.459755 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:56.459719    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:56.459941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:56.459768    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:56.459941 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:56.459843    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:07:56.459941 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:56.459928    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:56.459941 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:56.459936    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:07:57.45991792 +0000 UTC m=+222.841734703 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : configmap "kube-rbac-proxy" not found
Jul 09 16:07:56.460093 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:56.459962    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:07:57.459952462 +0000 UTC m=+222.841769228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:57.466230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:57.466178    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:57.466230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:57.466238    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:57.466670 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:57.466311    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:07:57.466670 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:57.466329    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:57.466670 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:57.466373    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:07:59.466360798 +0000 UTC m=+224.848177564 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:57.466670 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:57.466386    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:07:59.466378529 +0000 UTC m=+224.848195295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : configmap "kube-rbac-proxy" not found
Jul 09 16:07:57.979779 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:57.979752    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-stdmx_04b0d083-c5d4-4cc0-906d-c373319e1c97/dns-node-resolver/0.log"
Jul 09 16:07:58.580987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:58.580959    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44htx_bd8a93ba-1327-414e-9755-4c2d0ab8a9d8/node-ca/0.log"
Jul 09 16:07:59.480705 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:59.480658    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:59.480705 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:59.480716    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:07:59.480952 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:59.480785    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:07:59.480952 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:59.480820    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:59.480952 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:59.480847    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:08:03.480832776 +0000 UTC m=+228.862649545 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : configmap "kube-rbac-proxy" not found
Jul 09 16:07:59.480952 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:07:59.480864    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:08:03.480853052 +0000 UTC m=+228.862669817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:07:59.586449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:07:59.586417    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jj4z8_a7add5d7-9318-4518-ac96-eb8d62dfc423/serve-healthcheck-canary/0.log"
Jul 09 16:08:03.512893 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:03.512837    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:03.513342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:03.512942    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:03.513342 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:08:03.513000    2558 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:08:03.513342 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:08:03.513020    2558 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:08:03.513342 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:08:03.513074    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:08:11.513058198 +0000 UTC m=+236.894874965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:08:03.513342 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:08:03.513089    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm podName:1e5f1cbd-5b16-4b34-a88c-9de143e9c247 nodeName:}" failed. No retries permitted until 2026-07-09 16:08:11.513082881 +0000 UTC m=+236.894899647 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2x8xz" (UID: "1e5f1cbd-5b16-4b34-a88c-9de143e9c247") : configmap "kube-rbac-proxy" not found
Jul 09 16:08:11.574150 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.574115    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:11.574583 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.574162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:11.574627 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.574587    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:11.576503 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.576487    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1e5f1cbd-5b16-4b34-a88c-9de143e9c247-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2x8xz\" (UID: \"1e5f1cbd-5b16-4b34-a88c-9de143e9c247\") " pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:11.692192 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.692156    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2x8xz"
Jul 09 16:08:11.815895 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.815845    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2x8xz"]
Jul 09 16:08:11.819343 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:08:11.819316    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5f1cbd_5b16_4b34_a88c_9de143e9c247.slice/crio-58f93517fdcd37e06c1a664733888f82a9e3813647c980b8c47cc62e7f95150a WatchSource:0}: Error finding container 58f93517fdcd37e06c1a664733888f82a9e3813647c980b8c47cc62e7f95150a: Status 404 returned error can't find the container with id 58f93517fdcd37e06c1a664733888f82a9e3813647c980b8c47cc62e7f95150a
Jul 09 16:08:11.921271 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:11.921187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2x8xz" event={"ID":"1e5f1cbd-5b16-4b34-a88c-9de143e9c247","Type":"ContainerStarted","Data":"58f93517fdcd37e06c1a664733888f82a9e3813647c980b8c47cc62e7f95150a"}
Jul 09 16:08:16.942484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:16.942448    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2x8xz" event={"ID":"1e5f1cbd-5b16-4b34-a88c-9de143e9c247","Type":"ContainerStarted","Data":"14cd4321612f46ff376c4d3067c1bfab2c2c4ec99cd1d5e0a50ede754c988381"}
Jul 09 16:08:17.948731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:17.948689    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2x8xz" event={"ID":"1e5f1cbd-5b16-4b34-a88c-9de143e9c247","Type":"ContainerStarted","Data":"31de2b13faff15a75a0fa2ce8942e5ace74eb5ee6ca97705c1746881a4664d03"}
Jul 09 16:08:19.956462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:19.956423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2x8xz" event={"ID":"1e5f1cbd-5b16-4b34-a88c-9de143e9c247","Type":"ContainerStarted","Data":"b6fd38317ef00785d618406b6adb0d5468528d9ff77607a69b0b59c07158a513"}
Jul 09 16:08:19.975989 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:19.975942    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2x8xz" podStartSLOduration=17.142865439 podStartE2EDuration="24.975927814s" totalImagesPullingTime="7.833062375s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:07:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:08:19.974906253 +0000 UTC m=+245.356723043" watchObservedRunningTime="2026-07-09 16:08:19.975927814 +0000 UTC m=+245.357744601"
Jul 09 16:08:21.439862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.439823    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6674bc895c-zf727"]
Jul 09 16:08:21.442019 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.442000    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.445792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.445767    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 16:08:21.445944 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.445817    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nnrq4\""
Jul 09 16:08:21.446090 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.446075    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 16:08:21.447339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.447319    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 16:08:21.451419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.451402    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 16:08:21.460101 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.460074    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6674bc895c-zf727"]
Jul 09 16:08:21.542383 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542346    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-installation-pull-secrets\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542383 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542384    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-registry-certificates\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542625 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542439    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhnh\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-kube-api-access-hlhnh\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542625 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542486    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-trusted-ca\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542625 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542505    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-registry-tls\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542625 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542528    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-image-registry-private-configuration\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542625 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-bound-sa-token\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.542803 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.542626    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-ca-trust-extracted\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643551 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643514    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-bound-sa-token\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-ca-trust-extracted\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-installation-pull-secrets\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-registry-certificates\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhnh\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-kube-api-access-hlhnh\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643689    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-trusted-ca\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643711    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-registry-tls\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.643743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.643740    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-image-registry-private-configuration\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.644107 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.644030    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-ca-trust-extracted\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.644475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.644452    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-registry-certificates\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.644821 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.644599    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-trusted-ca\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.646366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.646349    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-image-registry-private-configuration\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.646480 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.646451    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-registry-tls\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.646561 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.646482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-installation-pull-secrets\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.659764 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.659734    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhnh\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-kube-api-access-hlhnh\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.660777 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.660749    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8000b1bb-e41b-4608-a2b4-a560d6fbbd69-bound-sa-token\") pod \"image-registry-6674bc895c-zf727\" (UID: \"8000b1bb-e41b-4608-a2b4-a560d6fbbd69\") " pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.751464 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.751364    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.881961 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.881931    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6674bc895c-zf727"]
Jul 09 16:08:21.884982 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:08:21.884953    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8000b1bb_e41b_4608_a2b4_a560d6fbbd69.slice/crio-4562419d760882e851b87f188079698a54d82072db20cd9dfba6935382069799 WatchSource:0}: Error finding container 4562419d760882e851b87f188079698a54d82072db20cd9dfba6935382069799: Status 404 returned error can't find the container with id 4562419d760882e851b87f188079698a54d82072db20cd9dfba6935382069799
Jul 09 16:08:21.964539 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.964488    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6674bc895c-zf727" event={"ID":"8000b1bb-e41b-4608-a2b4-a560d6fbbd69","Type":"ContainerStarted","Data":"9e9c2e401646f0627a3ef65195a59156c78995effd033c5de2e2fc3092b145bc"}
Jul 09 16:08:21.964725 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.964556    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6674bc895c-zf727" event={"ID":"8000b1bb-e41b-4608-a2b4-a560d6fbbd69","Type":"ContainerStarted","Data":"4562419d760882e851b87f188079698a54d82072db20cd9dfba6935382069799"}
Jul 09 16:08:21.964725 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.964630    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:21.986155 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:21.986102    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6674bc895c-zf727" podStartSLOduration=0.986083445 podStartE2EDuration="986.083445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:08:21.985104859 +0000 UTC m=+247.366921647" watchObservedRunningTime="2026-07-09 16:08:21.986083445 +0000 UTC m=+247.367900232"
Jul 09 16:08:26.976119 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:26.976081    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:08:26.978587 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:26.978566    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cca3ed8-8427-4ca7-a1af-7541a27782ff-metrics-certs\") pod \"network-metrics-daemon-6wzgz\" (UID: \"0cca3ed8-8427-4ca7-a1af-7541a27782ff\") " pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:08:27.275278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.275198    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gdktf\""
Jul 09 16:08:27.282692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.282660    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6wzgz"
Jul 09 16:08:27.407111 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.407080    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6wzgz"]
Jul 09 16:08:27.409614 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:08:27.409583    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cca3ed8_8427_4ca7_a1af_7541a27782ff.slice/crio-33c8a1b0c8c074ae269210b3156f6cc5f69b109f50cd3cca7d77e7dc7dd11718 WatchSource:0}: Error finding container 33c8a1b0c8c074ae269210b3156f6cc5f69b109f50cd3cca7d77e7dc7dd11718: Status 404 returned error can't find the container with id 33c8a1b0c8c074ae269210b3156f6cc5f69b109f50cd3cca7d77e7dc7dd11718
Jul 09 16:08:27.983417 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.983383    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wzgz" event={"ID":"0cca3ed8-8427-4ca7-a1af-7541a27782ff","Type":"ContainerStarted","Data":"33c8a1b0c8c074ae269210b3156f6cc5f69b109f50cd3cca7d77e7dc7dd11718"}
Jul 09 16:08:27.992807 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.992782    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jl9hb"]
Jul 09 16:08:27.995459 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.995440    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:27.998426 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.998402    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 16:08:27.998529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.998461    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 16:08:27.998529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.998496    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 16:08:27.998529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.998466    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 16:08:27.998768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:27.998752    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 16:08:28.000260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.000240    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 16:08:28.000363 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.000345    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cq2lf\""
Jul 09 16:08:28.082895 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.082855    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.082920    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-sys\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.082949    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtj9m\" (UniqueName: \"kubernetes.io/projected/4ab8077e-455d-4d43-94ec-b192d34a6d4c-kube-api-access-rtj9m\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.082988    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-wtmp\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.083015    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.083035    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-tls\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.083056    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-root\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083344 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.083085    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-textfile\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.083344 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.083128    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab8077e-455d-4d43-94ec-b192d34a6d4c-metrics-client-ca\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184079 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184035    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184086    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-sys\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184113    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtj9m\" (UniqueName: \"kubernetes.io/projected/4ab8077e-455d-4d43-94ec-b192d34a6d4c-kube-api-access-rtj9m\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-wtmp\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184195    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-tls\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184227    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-root\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184237    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-sys\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-textfile\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184659 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-wtmp\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184659 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab8077e-455d-4d43-94ec-b192d34a6d4c-metrics-client-ca\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184659 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184479    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4ab8077e-455d-4d43-94ec-b192d34a6d4c-root\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184659 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-textfile\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.184903 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184861    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4ab8077e-455d-4d43-94ec-b192d34a6d4c-metrics-client-ca\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.185019 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.184998    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-accelerators-collector-config\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.186546 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.186524    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.186648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.186632    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4ab8077e-455d-4d43-94ec-b192d34a6d4c-node-exporter-tls\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.195230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.195200    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtj9m\" (UniqueName: \"kubernetes.io/projected/4ab8077e-455d-4d43-94ec-b192d34a6d4c-kube-api-access-rtj9m\") pod \"node-exporter-jl9hb\" (UID: \"4ab8077e-455d-4d43-94ec-b192d34a6d4c\") " pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.305305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.305128    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jl9hb"
Jul 09 16:08:28.314493 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:08:28.314469    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab8077e_455d_4d43_94ec_b192d34a6d4c.slice/crio-4ca9969a4fde20ac888a01f46c8debff4870ab9a36513abb59a844eddc97198a WatchSource:0}: Error finding container 4ca9969a4fde20ac888a01f46c8debff4870ab9a36513abb59a844eddc97198a: Status 404 returned error can't find the container with id 4ca9969a4fde20ac888a01f46c8debff4870ab9a36513abb59a844eddc97198a
Jul 09 16:08:28.988997 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:28.988962    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jl9hb" event={"ID":"4ab8077e-455d-4d43-94ec-b192d34a6d4c","Type":"ContainerStarted","Data":"4ca9969a4fde20ac888a01f46c8debff4870ab9a36513abb59a844eddc97198a"}
Jul 09 16:08:29.993779 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:29.993740    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wzgz" event={"ID":"0cca3ed8-8427-4ca7-a1af-7541a27782ff","Type":"ContainerStarted","Data":"ff699c87696653e738f6a296226618e5d3d082dc833e12eea12a0e6728c6242e"}
Jul 09 16:08:29.994250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:29.993847    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6wzgz" event={"ID":"0cca3ed8-8427-4ca7-a1af-7541a27782ff","Type":"ContainerStarted","Data":"689bb8f636f2758987286c910b7a3413d385d3a63c1e613eb9cb50fde1283eed"}
Jul 09 16:08:29.995318 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:29.995295    2558 generic.go:356] "Generic (PLEG): container finished" podID="4ab8077e-455d-4d43-94ec-b192d34a6d4c" containerID="95bf8af1c943b76373e19bf410505ee68876fe358e373b54792960704440c365" exitCode=0
Jul 09 16:08:29.995417 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:29.995325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jl9hb" event={"ID":"4ab8077e-455d-4d43-94ec-b192d34a6d4c","Type":"ContainerDied","Data":"95bf8af1c943b76373e19bf410505ee68876fe358e373b54792960704440c365"}
Jul 09 16:08:30.037166 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:30.037122    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6wzgz" podStartSLOduration=253.195863516 podStartE2EDuration="4m15.037107426s" totalImagesPullingTime="1.84124391s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:08:30.036226566 +0000 UTC m=+255.418043357" watchObservedRunningTime="2026-07-09 16:08:30.037107426 +0000 UTC m=+255.418924215"
Jul 09 16:08:31.000814 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:31.000777    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jl9hb" event={"ID":"4ab8077e-455d-4d43-94ec-b192d34a6d4c","Type":"ContainerStarted","Data":"ac68e2413f1475e9b6ef452e54903d53a29199b2d9be7c5b0085a4b0883e276e"}
Jul 09 16:08:31.001295 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:31.000842    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jl9hb" event={"ID":"4ab8077e-455d-4d43-94ec-b192d34a6d4c","Type":"ContainerStarted","Data":"5e72dd6fe501faa8c3b0358048286f3fbe2c023f20c1d65d592be2315c523056"}
Jul 09 16:08:31.023764 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:31.023717    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jl9hb" podStartSLOduration=3.009863537 podStartE2EDuration="4.023702708s" totalImagesPullingTime="937.31255ms" totalInitContainerRuntime="76.526621ms" isStatefulPod=true podCreationTimestamp="2026-07-09 16:08:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:08:31.021935767 +0000 UTC m=+256.403752556" watchObservedRunningTime="2026-07-09 16:08:31.023702708 +0000 UTC m=+256.405519495"
Jul 09 16:08:32.219493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.219460    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6bf65bc649-r2mk2"]
Jul 09 16:08:32.221511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.221484    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.226027 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.225998    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 09 16:08:32.226027 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.226019    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 09 16:08:32.226204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.226039    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 09 16:08:32.226204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.226083    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 09 16:08:32.226311 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.226230    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3hmluudvvt2tg\""
Jul 09 16:08:32.226311 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.226268    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-x6774\""
Jul 09 16:08:32.231206 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.231187    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bf65bc649-r2mk2"]
Jul 09 16:08:32.314939 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.314895    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-secret-metrics-server-tls\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.315089 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.314966    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/355658df-4653-4828-be43-cd8b3e8b0135-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.315089 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.314992    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/355658df-4653-4828-be43-cd8b3e8b0135-metrics-server-audit-profiles\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.315089 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.315014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-client-ca-bundle\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.315089 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.315036    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhdk\" (UniqueName: \"kubernetes.io/projected/355658df-4653-4828-be43-cd8b3e8b0135-kube-api-access-sdhdk\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.315236 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.315106    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-secret-metrics-server-client-certs\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.315236 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.315137    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/355658df-4653-4828-be43-cd8b3e8b0135-audit-log\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415579 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415544    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-secret-metrics-server-client-certs\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415586    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/355658df-4653-4828-be43-cd8b3e8b0135-audit-log\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415619    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-secret-metrics-server-tls\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415666    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/355658df-4653-4828-be43-cd8b3e8b0135-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415714    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/355658df-4653-4828-be43-cd8b3e8b0135-metrics-server-audit-profiles\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415944 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415762    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-client-ca-bundle\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.415944 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.415784    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhdk\" (UniqueName: \"kubernetes.io/projected/355658df-4653-4828-be43-cd8b3e8b0135-kube-api-access-sdhdk\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.416111 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.416086    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/355658df-4653-4828-be43-cd8b3e8b0135-audit-log\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.416484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.416462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/355658df-4653-4828-be43-cd8b3e8b0135-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.417004 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.416981    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/355658df-4653-4828-be43-cd8b3e8b0135-metrics-server-audit-profiles\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.418241 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.418218    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-secret-metrics-server-client-certs\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.418415 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.418392    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-client-ca-bundle\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.418505 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.418486    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/355658df-4653-4828-be43-cd8b3e8b0135-secret-metrics-server-tls\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.424332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.424313    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhdk\" (UniqueName: \"kubernetes.io/projected/355658df-4653-4828-be43-cd8b3e8b0135-kube-api-access-sdhdk\") pod \"metrics-server-6bf65bc649-r2mk2\" (UID: \"355658df-4653-4828-be43-cd8b3e8b0135\") " pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.531221 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.531143    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:32.666043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:32.666016    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6bf65bc649-r2mk2"]
Jul 09 16:08:32.671289 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:08:32.669572    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod355658df_4653_4828_be43_cd8b3e8b0135.slice/crio-812f17abc5eaa29070ebddd02f0717dfd523a15d20207eaced2983f974a3d201 WatchSource:0}: Error finding container 812f17abc5eaa29070ebddd02f0717dfd523a15d20207eaced2983f974a3d201: Status 404 returned error can't find the container with id 812f17abc5eaa29070ebddd02f0717dfd523a15d20207eaced2983f974a3d201
Jul 09 16:08:33.012272 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:33.012235    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2" event={"ID":"355658df-4653-4828-be43-cd8b3e8b0135","Type":"ContainerStarted","Data":"812f17abc5eaa29070ebddd02f0717dfd523a15d20207eaced2983f974a3d201"}
Jul 09 16:08:35.021055 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:35.020973    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2" event={"ID":"355658df-4653-4828-be43-cd8b3e8b0135","Type":"ContainerStarted","Data":"f33c0d09f4ef4a34d3c80790f4f71fcd63d3c267d93ba5bc519e7ec0ba68479e"}
Jul 09 16:08:35.041558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:35.041492    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2" podStartSLOduration=1.5337947170000001 podStartE2EDuration="3.041478697s" totalImagesPullingTime="1.50768398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:08:35.039820802 +0000 UTC m=+260.421637590" watchObservedRunningTime="2026-07-09 16:08:35.041478697 +0000 UTC m=+260.423295486"
Jul 09 16:08:41.755507 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:41.755470    2558 patch_prober.go:28] interesting pod/image-registry-6674bc895c-zf727 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 16:08:41.755866 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:41.755529    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6674bc895c-zf727" podUID="8000b1bb-e41b-4608-a2b4-a560d6fbbd69" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:08:42.972159 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:42.972126    2558 patch_prober.go:28] interesting pod/image-registry-6674bc895c-zf727 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 16:08:42.972662 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:42.972185    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6674bc895c-zf727" podUID="8000b1bb-e41b-4608-a2b4-a560d6fbbd69" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:08:51.755998 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:51.755966    2558 patch_prober.go:28] interesting pod/image-registry-6674bc895c-zf727 container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 16:08:51.756427 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:51.756024    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6674bc895c-zf727" podUID="8000b1bb-e41b-4608-a2b4-a560d6fbbd69" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:08:52.531576 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:52.531537    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:52.531747 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:52.531623    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:08:52.971822 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:52.971791    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6674bc895c-zf727"
Jul 09 16:08:56.701602 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:08:56.701558    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fhktg" podUID="36639cf2-b21a-4d83-b9ba-e07f0412759d"
Jul 09 16:08:57.089991 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:08:57.089961    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fhktg"
Jul 09 16:09:00.420819 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:00.420783    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:09:00.423116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:00.423096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36639cf2-b21a-4d83-b9ba-e07f0412759d-metrics-tls\") pod \"dns-default-fhktg\" (UID: \"36639cf2-b21a-4d83-b9ba-e07f0412759d\") " pod="openshift-dns/dns-default-fhktg"
Jul 09 16:09:00.693644 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:00.693561    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tvt5b\""
Jul 09 16:09:00.701120 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:00.701094    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fhktg"
Jul 09 16:09:00.869271 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:00.869236    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fhktg"]
Jul 09 16:09:00.872087 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:09:00.872057    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36639cf2_b21a_4d83_b9ba_e07f0412759d.slice/crio-458b6e7147d881d72eb1b323574c5462a93068dbf199f4fca62463dac2102707 WatchSource:0}: Error finding container 458b6e7147d881d72eb1b323574c5462a93068dbf199f4fca62463dac2102707: Status 404 returned error can't find the container with id 458b6e7147d881d72eb1b323574c5462a93068dbf199f4fca62463dac2102707
Jul 09 16:09:01.103250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:01.103215    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fhktg" event={"ID":"36639cf2-b21a-4d83-b9ba-e07f0412759d","Type":"ContainerStarted","Data":"458b6e7147d881d72eb1b323574c5462a93068dbf199f4fca62463dac2102707"}
Jul 09 16:09:03.111339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:03.111305    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fhktg" event={"ID":"36639cf2-b21a-4d83-b9ba-e07f0412759d","Type":"ContainerStarted","Data":"d777de92481692491c38053717def1b388ffab00f79e2dd0ad9b5c589b3b8544"}
Jul 09 16:09:03.111710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:03.111357    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fhktg" event={"ID":"36639cf2-b21a-4d83-b9ba-e07f0412759d","Type":"ContainerStarted","Data":"86201bdaa1bbb650074ed56a1b01fec9dc02720369bff25802857ccf963cb629"}
Jul 09 16:09:03.111710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:03.111450    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fhktg"
Jul 09 16:09:03.138652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:03.138606    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fhktg" podStartSLOduration=251.102528304 podStartE2EDuration="4m13.138591444s" totalImagesPullingTime="2.03606314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:09:03.137195202 +0000 UTC m=+288.519011991" watchObservedRunningTime="2026-07-09 16:09:03.138591444 +0000 UTC m=+288.520408231"
Jul 09 16:09:06.325184 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:06.325149    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jj4z8_a7add5d7-9318-4518-ac96-eb8d62dfc423/serve-healthcheck-canary/0.log"
Jul 09 16:09:12.537559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:12.537526    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:09:12.542975 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:12.542952    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6bf65bc649-r2mk2"
Jul 09 16:09:13.116766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:13.116740    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fhktg"
Jul 09 16:09:13.923418 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:13.923377    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" podUID="45e82465-f8ac-4c6e-ba5f-b201f5b0c118" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 16:09:14.982907 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:14.982862    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:09:14.984041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:14.984021    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:09:14.986705 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:14.986689    2558 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 16:09:23.923431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:23.923394    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" podUID="45e82465-f8ac-4c6e-ba5f-b201f5b0c118" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 16:09:33.923433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:33.923386    2558 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" podUID="45e82465-f8ac-4c6e-ba5f-b201f5b0c118" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 09 16:09:33.923824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:33.923462    2558 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7"
Jul 09 16:09:33.923961 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:33.923932    2558 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"2de1ff7d05ea3c36834950a886f3d0182c32bb1d2467b4e8b120474ac9d1458f"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 09 16:09:33.924007 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:33.923984    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" podUID="45e82465-f8ac-4c6e-ba5f-b201f5b0c118" containerName="service-proxy" containerID="cri-o://2de1ff7d05ea3c36834950a886f3d0182c32bb1d2467b4e8b120474ac9d1458f" gracePeriod=30
Jul 09 16:09:34.210948 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:34.210852    2558 generic.go:356] "Generic (PLEG): container finished" podID="45e82465-f8ac-4c6e-ba5f-b201f5b0c118" containerID="2de1ff7d05ea3c36834950a886f3d0182c32bb1d2467b4e8b120474ac9d1458f" exitCode=2
Jul 09 16:09:34.210948 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:34.210913    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" event={"ID":"45e82465-f8ac-4c6e-ba5f-b201f5b0c118","Type":"ContainerDied","Data":"2de1ff7d05ea3c36834950a886f3d0182c32bb1d2467b4e8b120474ac9d1458f"}
Jul 09 16:09:34.210948 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:09:34.210949    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b4d969697-z6vv7" event={"ID":"45e82465-f8ac-4c6e-ba5f-b201f5b0c118","Type":"ContainerStarted","Data":"be48991f7a2f12ea08c9a1286b3c98b6cc45af2febfb0fd63b824b5018696f0b"}
Jul 09 16:12:20.424848 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.424810    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"]
Jul 09 16:12:20.427214 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.427197    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.430475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.430451    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 16:12:20.431705 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.431685    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 16:12:20.431843 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.431692    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7bjxc\""
Jul 09 16:12:20.439556 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.439536    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"]
Jul 09 16:12:20.525365 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.525321    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.525542 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.525375    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.525542 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.525435    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5w9m\" (UniqueName: \"kubernetes.io/projected/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-kube-api-access-h5w9m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.626721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.626614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.626869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.626775    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.626869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.626806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5w9m\" (UniqueName: \"kubernetes.io/projected/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-kube-api-access-h5w9m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.626972 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.626948    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.627143 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.627123    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.636332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.636303    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5w9m\" (UniqueName: \"kubernetes.io/projected/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-kube-api-access-h5w9m\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.735831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.735749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:20.864142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.864102    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"]
Jul 09 16:12:20.867022 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:12:20.866998    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fe2715_bc5f_4d8d_a72b_7b4b8b3615e4.slice/crio-86855aaf3623ebb00c75e12cdd5fd1f46f2b480cf1a33e2ae4a07c30c1ba1b30 WatchSource:0}: Error finding container 86855aaf3623ebb00c75e12cdd5fd1f46f2b480cf1a33e2ae4a07c30c1ba1b30: Status 404 returned error can't find the container with id 86855aaf3623ebb00c75e12cdd5fd1f46f2b480cf1a33e2ae4a07c30c1ba1b30
Jul 09 16:12:20.868724 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:20.868709    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:12:21.752965 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:21.752934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh" event={"ID":"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4","Type":"ContainerStarted","Data":"86855aaf3623ebb00c75e12cdd5fd1f46f2b480cf1a33e2ae4a07c30c1ba1b30"}
Jul 09 16:12:35.800328 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:35.800294    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerID="28bf78b8df4e1071c234e4c325975db94047be1b86cd2147473e896c828d9d36" exitCode=0
Jul 09 16:12:35.800803 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:35.800341    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh" event={"ID":"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4","Type":"ContainerDied","Data":"28bf78b8df4e1071c234e4c325975db94047be1b86cd2147473e896c828d9d36"}
Jul 09 16:12:40.820433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:40.820397    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerID="cf2cb72e0b8e8ba6428edb080fb8e5de6a46536879d5da6eb62ded38eff24685" exitCode=0
Jul 09 16:12:40.820897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:40.820479    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh" event={"ID":"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4","Type":"ContainerDied","Data":"cf2cb72e0b8e8ba6428edb080fb8e5de6a46536879d5da6eb62ded38eff24685"}
Jul 09 16:12:51.860652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:51.860615    2558 generic.go:356] "Generic (PLEG): container finished" podID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerID="d6b23c2f9493eddfa13f8ab818c639e53210bb08032537b40f12df043ee47266" exitCode=0
Jul 09 16:12:51.861044 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:51.860690    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh" event={"ID":"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4","Type":"ContainerDied","Data":"d6b23c2f9493eddfa13f8ab818c639e53210bb08032537b40f12df043ee47266"}
Jul 09 16:12:52.985406 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:52.985381    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:53.076104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.076072    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5w9m\" (UniqueName: \"kubernetes.io/projected/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-kube-api-access-h5w9m\") pod \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") "
Jul 09 16:12:53.076300 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.076121    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-bundle\") pod \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") "
Jul 09 16:12:53.076300 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.076158    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-util\") pod \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\" (UID: \"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4\") "
Jul 09 16:12:53.076852 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.076817    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-bundle" (OuterVolumeSpecName: "bundle") pod "e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" (UID: "e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:12:53.078424 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.078396    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-kube-api-access-h5w9m" (OuterVolumeSpecName: "kube-api-access-h5w9m") pod "e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" (UID: "e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4"). InnerVolumeSpecName "kube-api-access-h5w9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:12:53.081183 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.081164    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-util" (OuterVolumeSpecName: "util") pod "e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" (UID: "e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:12:53.176772 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.176698    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5w9m\" (UniqueName: \"kubernetes.io/projected/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-kube-api-access-h5w9m\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:12:53.176772 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.176729    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:12:53.176772 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.176742    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:12:53.869301 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.869275    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh"
Jul 09 16:12:53.869458 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.869273    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540gjbrh" event={"ID":"e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4","Type":"ContainerDied","Data":"86855aaf3623ebb00c75e12cdd5fd1f46f2b480cf1a33e2ae4a07c30c1ba1b30"}
Jul 09 16:12:53.869458 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:53.869392    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86855aaf3623ebb00c75e12cdd5fd1f46f2b480cf1a33e2ae4a07c30c1ba1b30"
Jul 09 16:12:58.570259 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570223    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"]
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570533    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="util"
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570551    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="util"
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570569    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="pull"
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570576    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="pull"
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570590    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="extract"
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570598    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="extract"
Jul 09 16:12:58.570818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.570658    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2fe2715-bc5f-4d8d-a72b-7b4b8b3615e4" containerName="extract"
Jul 09 16:12:58.574378 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.574357    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.577946 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.577925    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 09 16:12:58.578053 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.578003    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-vx6fl\""
Jul 09 16:12:58.579046 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.579026    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:12:58.593072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.593044    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"]
Jul 09 16:12:58.715198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.715163    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9zlq\" (UniqueName: \"kubernetes.io/projected/8499d2bb-6d5c-40e0-83e7-dd75c9eff944-kube-api-access-b9zlq\") pod \"cert-manager-operator-controller-manager-55c444bf86-tsc7q\" (UID: \"8499d2bb-6d5c-40e0-83e7-dd75c9eff944\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.715198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.715204    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8499d2bb-6d5c-40e0-83e7-dd75c9eff944-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tsc7q\" (UID: \"8499d2bb-6d5c-40e0-83e7-dd75c9eff944\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.815984 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.815950    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9zlq\" (UniqueName: \"kubernetes.io/projected/8499d2bb-6d5c-40e0-83e7-dd75c9eff944-kube-api-access-b9zlq\") pod \"cert-manager-operator-controller-manager-55c444bf86-tsc7q\" (UID: \"8499d2bb-6d5c-40e0-83e7-dd75c9eff944\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.815984 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.815987    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8499d2bb-6d5c-40e0-83e7-dd75c9eff944-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tsc7q\" (UID: \"8499d2bb-6d5c-40e0-83e7-dd75c9eff944\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.816310 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.816295    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8499d2bb-6d5c-40e0-83e7-dd75c9eff944-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-tsc7q\" (UID: \"8499d2bb-6d5c-40e0-83e7-dd75c9eff944\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.825947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.825866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9zlq\" (UniqueName: \"kubernetes.io/projected/8499d2bb-6d5c-40e0-83e7-dd75c9eff944-kube-api-access-b9zlq\") pod \"cert-manager-operator-controller-manager-55c444bf86-tsc7q\" (UID: \"8499d2bb-6d5c-40e0-83e7-dd75c9eff944\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:58.883421 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:58.883391    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"
Jul 09 16:12:59.010718 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:59.010695    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q"]
Jul 09 16:12:59.013299 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:12:59.013274    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8499d2bb_6d5c_40e0_83e7_dd75c9eff944.slice/crio-054bdb9447c75df8942be018d02613b835813da5faedf6d65432eecdafe7713f WatchSource:0}: Error finding container 054bdb9447c75df8942be018d02613b835813da5faedf6d65432eecdafe7713f: Status 404 returned error can't find the container with id 054bdb9447c75df8942be018d02613b835813da5faedf6d65432eecdafe7713f
Jul 09 16:12:59.891606 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:12:59.891570    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q" event={"ID":"8499d2bb-6d5c-40e0-83e7-dd75c9eff944","Type":"ContainerStarted","Data":"054bdb9447c75df8942be018d02613b835813da5faedf6d65432eecdafe7713f"}
Jul 09 16:13:00.897381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:00.897340    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q" event={"ID":"8499d2bb-6d5c-40e0-83e7-dd75c9eff944","Type":"ContainerStarted","Data":"17c8c3fdae3b9c8825e93fab512093b7e2c160f84513ccc8b019a4cec2941d12"}
Jul 09 16:13:00.923217 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:00.922521    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-tsc7q" podStartSLOduration=1.459665061 podStartE2EDuration="2.92249995s" totalImagesPullingTime="1.462834889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:13:00.91969614 +0000 UTC m=+526.301512929" watchObservedRunningTime="2026-07-09 16:13:00.92249995 +0000 UTC m=+526.304316740"
Jul 09 16:13:02.218269 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.218232    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"]
Jul 09 16:13:02.220645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.220623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.223442 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.223422    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7bjxc\""
Jul 09 16:13:02.223518 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.223429    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 16:13:02.224863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.224848    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 16:13:02.233459 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.233441    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"]
Jul 09 16:13:02.346440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.346408    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.346571 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.346447    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkjw2\" (UniqueName: \"kubernetes.io/projected/0c19929e-4386-4644-8c0b-dc091908d743-kube-api-access-pkjw2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.346571 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.346522    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.447560 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.447525    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.447560 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.447561    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkjw2\" (UniqueName: \"kubernetes.io/projected/0c19929e-4386-4644-8c0b-dc091908d743-kube-api-access-pkjw2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.447753 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.447605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.447873 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.447853    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.447995 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.447976    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.462429 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.462402    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkjw2\" (UniqueName: \"kubernetes.io/projected/0c19929e-4386-4644-8c0b-dc091908d743-kube-api-access-pkjw2\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.528974 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.528910    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:02.655375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.655347    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"]
Jul 09 16:13:02.658274 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:13:02.658243    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c19929e_4386_4644_8c0b_dc091908d743.slice/crio-1fabf849afc7da5c6eff90f6a8b0e1bc8f8163662da24a2f8e759295a6f1ad0f WatchSource:0}: Error finding container 1fabf849afc7da5c6eff90f6a8b0e1bc8f8163662da24a2f8e759295a6f1ad0f: Status 404 returned error can't find the container with id 1fabf849afc7da5c6eff90f6a8b0e1bc8f8163662da24a2f8e759295a6f1ad0f
Jul 09 16:13:02.906453 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.906421    2558 generic.go:356] "Generic (PLEG): container finished" podID="0c19929e-4386-4644-8c0b-dc091908d743" containerID="f8346613d7a78bc12af398b80311808e6d09fea385634bc60cf6aa82face8430" exitCode=0
Jul 09 16:13:02.906619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.906466    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl" event={"ID":"0c19929e-4386-4644-8c0b-dc091908d743","Type":"ContainerDied","Data":"f8346613d7a78bc12af398b80311808e6d09fea385634bc60cf6aa82face8430"}
Jul 09 16:13:02.906619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:02.906501    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl" event={"ID":"0c19929e-4386-4644-8c0b-dc091908d743","Type":"ContainerStarted","Data":"1fabf849afc7da5c6eff90f6a8b0e1bc8f8163662da24a2f8e759295a6f1ad0f"}
Jul 09 16:13:05.921269 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:05.921228    2558 generic.go:356] "Generic (PLEG): container finished" podID="0c19929e-4386-4644-8c0b-dc091908d743" containerID="8b8843b1ec41ddf4d5e62fc1ad6bc8b9d9754b635c1dd51bf9598eede526132d" exitCode=0
Jul 09 16:13:05.921655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:05.921272    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl" event={"ID":"0c19929e-4386-4644-8c0b-dc091908d743","Type":"ContainerDied","Data":"8b8843b1ec41ddf4d5e62fc1ad6bc8b9d9754b635c1dd51bf9598eede526132d"}
Jul 09 16:13:06.927371 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:06.927341    2558 generic.go:356] "Generic (PLEG): container finished" podID="0c19929e-4386-4644-8c0b-dc091908d743" containerID="16ad51f42cf01f3437ddeac08326543f3fb548b25b53ec5bf381bc30367269ef" exitCode=0
Jul 09 16:13:06.927784 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:06.927421    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl" event={"ID":"0c19929e-4386-4644-8c0b-dc091908d743","Type":"ContainerDied","Data":"16ad51f42cf01f3437ddeac08326543f3fb548b25b53ec5bf381bc30367269ef"}
Jul 09 16:13:08.053678 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.053655    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:08.086813 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.086777    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-bundle\") pod \"0c19929e-4386-4644-8c0b-dc091908d743\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") "
Jul 09 16:13:08.086998 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.086831    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkjw2\" (UniqueName: \"kubernetes.io/projected/0c19929e-4386-4644-8c0b-dc091908d743-kube-api-access-pkjw2\") pod \"0c19929e-4386-4644-8c0b-dc091908d743\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") "
Jul 09 16:13:08.086998 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.086901    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-util\") pod \"0c19929e-4386-4644-8c0b-dc091908d743\" (UID: \"0c19929e-4386-4644-8c0b-dc091908d743\") "
Jul 09 16:13:08.087281 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.087256    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-bundle" (OuterVolumeSpecName: "bundle") pod "0c19929e-4386-4644-8c0b-dc091908d743" (UID: "0c19929e-4386-4644-8c0b-dc091908d743"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:13:08.089185 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.089149    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c19929e-4386-4644-8c0b-dc091908d743-kube-api-access-pkjw2" (OuterVolumeSpecName: "kube-api-access-pkjw2") pod "0c19929e-4386-4644-8c0b-dc091908d743" (UID: "0c19929e-4386-4644-8c0b-dc091908d743"). InnerVolumeSpecName "kube-api-access-pkjw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:13:08.092170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.092141    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-util" (OuterVolumeSpecName: "util") pod "0c19929e-4386-4644-8c0b-dc091908d743" (UID: "0c19929e-4386-4644-8c0b-dc091908d743"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:13:08.187841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.187738    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:13:08.187841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.187779    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c19929e-4386-4644-8c0b-dc091908d743-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:13:08.188098 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.187867    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkjw2\" (UniqueName: \"kubernetes.io/projected/0c19929e-4386-4644-8c0b-dc091908d743-kube-api-access-pkjw2\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:13:08.936414 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.936382    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl" event={"ID":"0c19929e-4386-4644-8c0b-dc091908d743","Type":"ContainerDied","Data":"1fabf849afc7da5c6eff90f6a8b0e1bc8f8163662da24a2f8e759295a6f1ad0f"}
Jul 09 16:13:08.936583 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.936427    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fabf849afc7da5c6eff90f6a8b0e1bc8f8163662da24a2f8e759295a6f1ad0f"
Jul 09 16:13:08.936583 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:08.936428    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87ftq9jl"
Jul 09 16:13:15.200511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200476    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"]
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200723    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="util"
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200733    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="util"
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200743    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="extract"
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200749    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="extract"
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200757    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="pull"
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200763    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="pull"
Jul 09 16:13:15.200936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.200808    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c19929e-4386-4644-8c0b-dc091908d743" containerName="extract"
Jul 09 16:13:15.203464 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.203449    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.206260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.206243    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:13:15.206369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.206283    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 16:13:15.207643 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.207626    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-js4dh\""
Jul 09 16:13:15.211760 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.211741    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"]
Jul 09 16:13:15.237441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.237404    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a205263e-e0d7-44d9-8a63-a03aa53b23d0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ps4kj\" (UID: \"a205263e-e0d7-44d9-8a63-a03aa53b23d0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.237612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.237467    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljhvv\" (UniqueName: \"kubernetes.io/projected/a205263e-e0d7-44d9-8a63-a03aa53b23d0-kube-api-access-ljhvv\") pod \"openshift-lws-operator-fd8ccff4c-ps4kj\" (UID: \"a205263e-e0d7-44d9-8a63-a03aa53b23d0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.338552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.338523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljhvv\" (UniqueName: \"kubernetes.io/projected/a205263e-e0d7-44d9-8a63-a03aa53b23d0-kube-api-access-ljhvv\") pod \"openshift-lws-operator-fd8ccff4c-ps4kj\" (UID: \"a205263e-e0d7-44d9-8a63-a03aa53b23d0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.338696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.338570    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a205263e-e0d7-44d9-8a63-a03aa53b23d0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ps4kj\" (UID: \"a205263e-e0d7-44d9-8a63-a03aa53b23d0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.338958 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.338942    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a205263e-e0d7-44d9-8a63-a03aa53b23d0-tmp\") pod \"openshift-lws-operator-fd8ccff4c-ps4kj\" (UID: \"a205263e-e0d7-44d9-8a63-a03aa53b23d0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.346943 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.346915    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljhvv\" (UniqueName: \"kubernetes.io/projected/a205263e-e0d7-44d9-8a63-a03aa53b23d0-kube-api-access-ljhvv\") pod \"openshift-lws-operator-fd8ccff4c-ps4kj\" (UID: \"a205263e-e0d7-44d9-8a63-a03aa53b23d0\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.512417 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.512320    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"
Jul 09 16:13:15.636668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.636628    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj"]
Jul 09 16:13:15.639496 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:13:15.639455    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda205263e_e0d7_44d9_8a63_a03aa53b23d0.slice/crio-e1a27bc1de87167e92a35b9b7c091541ea3599c8d00f3333f44af26806699f7b WatchSource:0}: Error finding container e1a27bc1de87167e92a35b9b7c091541ea3599c8d00f3333f44af26806699f7b: Status 404 returned error can't find the container with id e1a27bc1de87167e92a35b9b7c091541ea3599c8d00f3333f44af26806699f7b
Jul 09 16:13:15.962670 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:15.962635    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj" event={"ID":"a205263e-e0d7-44d9-8a63-a03aa53b23d0","Type":"ContainerStarted","Data":"e1a27bc1de87167e92a35b9b7c091541ea3599c8d00f3333f44af26806699f7b"}
Jul 09 16:13:17.972839 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:17.972804    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj" event={"ID":"a205263e-e0d7-44d9-8a63-a03aa53b23d0","Type":"ContainerStarted","Data":"7d76543a869f215dec3259e06359c163bd30d8bd49f05bec350116066ddac050"}
Jul 09 16:13:17.991831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:17.991781    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-ps4kj" podStartSLOduration=1.426438135 podStartE2EDuration="2.991765402s" totalImagesPullingTime="1.565327267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:13:17.991050345 +0000 UTC m=+543.372867132" watchObservedRunningTime="2026-07-09 16:13:17.991765402 +0000 UTC m=+543.373582187"
Jul 09 16:13:22.852521 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.852487    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"]
Jul 09 16:13:22.873565 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.873534    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"]
Jul 09 16:13:22.873732 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.873638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:22.876991 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.876967    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 09 16:13:22.877118 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.876999    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 09 16:13:22.877118 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.877037    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7bjxc\""
Jul 09 16:13:22.999419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.999392    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:22.999575 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.999443    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:22.999575 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:22.999501    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klg6z\" (UniqueName: \"kubernetes.io/projected/db315d18-fe3b-4130-88db-118ffa42b534-kube-api-access-klg6z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.100717 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.100685    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.100917 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.100732    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.100917 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.100767    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klg6z\" (UniqueName: \"kubernetes.io/projected/db315d18-fe3b-4130-88db-118ffa42b534-kube-api-access-klg6z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.101098 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.101080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.101154 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.101124    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.109331 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.109278    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klg6z\" (UniqueName: \"kubernetes.io/projected/db315d18-fe3b-4130-88db-118ffa42b534-kube-api-access-klg6z\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.183198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.183172    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:23.332158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.332134    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"]
Jul 09 16:13:23.334615 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:13:23.334583    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb315d18_fe3b_4130_88db_118ffa42b534.slice/crio-db384d64a3e65522d1f7db90c8c06d0995aa52302843baa21fb71417a3a53895 WatchSource:0}: Error finding container db384d64a3e65522d1f7db90c8c06d0995aa52302843baa21fb71417a3a53895: Status 404 returned error can't find the container with id db384d64a3e65522d1f7db90c8c06d0995aa52302843baa21fb71417a3a53895
Jul 09 16:13:23.997724 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.997690    2558 generic.go:356] "Generic (PLEG): container finished" podID="db315d18-fe3b-4130-88db-118ffa42b534" containerID="8e8259ca542e922395c6575b3957470de64656bb6850fe156e89f1c1ee8e3731" exitCode=0
Jul 09 16:13:23.998109 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.997785    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46" event={"ID":"db315d18-fe3b-4130-88db-118ffa42b534","Type":"ContainerDied","Data":"8e8259ca542e922395c6575b3957470de64656bb6850fe156e89f1c1ee8e3731"}
Jul 09 16:13:23.998109 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:23.997838    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46" event={"ID":"db315d18-fe3b-4130-88db-118ffa42b534","Type":"ContainerStarted","Data":"db384d64a3e65522d1f7db90c8c06d0995aa52302843baa21fb71417a3a53895"}
Jul 09 16:13:26.006321 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:26.006285    2558 generic.go:356] "Generic (PLEG): container finished" podID="db315d18-fe3b-4130-88db-118ffa42b534" containerID="a4e91c76f46dd93d138682ba05d3ac4a9b6b37862a517e05c140a94a91f92f45" exitCode=0
Jul 09 16:13:26.006759 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:26.006325    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46" event={"ID":"db315d18-fe3b-4130-88db-118ffa42b534","Type":"ContainerDied","Data":"a4e91c76f46dd93d138682ba05d3ac4a9b6b37862a517e05c140a94a91f92f45"}
Jul 09 16:13:27.012179 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:27.012145    2558 generic.go:356] "Generic (PLEG): container finished" podID="db315d18-fe3b-4130-88db-118ffa42b534" containerID="471c69a1f3a52db3f6279ac41e5e92ca9c4c1b938c4aa2b97f1cdc78e86ea69d" exitCode=0
Jul 09 16:13:27.012554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:27.012212    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46" event={"ID":"db315d18-fe3b-4130-88db-118ffa42b534","Type":"ContainerDied","Data":"471c69a1f3a52db3f6279ac41e5e92ca9c4c1b938c4aa2b97f1cdc78e86ea69d"}
Jul 09 16:13:28.136043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.136016    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:28.237233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.237205    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-util\") pod \"db315d18-fe3b-4130-88db-118ffa42b534\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") "
Jul 09 16:13:28.237367 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.237266    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-klg6z\" (UniqueName: \"kubernetes.io/projected/db315d18-fe3b-4130-88db-118ffa42b534-kube-api-access-klg6z\") pod \"db315d18-fe3b-4130-88db-118ffa42b534\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") "
Jul 09 16:13:28.237367 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.237294    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-bundle\") pod \"db315d18-fe3b-4130-88db-118ffa42b534\" (UID: \"db315d18-fe3b-4130-88db-118ffa42b534\") "
Jul 09 16:13:28.238032 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.238007    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-bundle" (OuterVolumeSpecName: "bundle") pod "db315d18-fe3b-4130-88db-118ffa42b534" (UID: "db315d18-fe3b-4130-88db-118ffa42b534"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:13:28.239285 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.239264    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db315d18-fe3b-4130-88db-118ffa42b534-kube-api-access-klg6z" (OuterVolumeSpecName: "kube-api-access-klg6z") pod "db315d18-fe3b-4130-88db-118ffa42b534" (UID: "db315d18-fe3b-4130-88db-118ffa42b534"). InnerVolumeSpecName "kube-api-access-klg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:13:28.242591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.242570    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-util" (OuterVolumeSpecName: "util") pod "db315d18-fe3b-4130-88db-118ffa42b534" (UID: "db315d18-fe3b-4130-88db-118ffa42b534"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:13:28.337706 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.337684    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klg6z\" (UniqueName: \"kubernetes.io/projected/db315d18-fe3b-4130-88db-118ffa42b534-kube-api-access-klg6z\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:13:28.337706 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.337706    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:13:28.337824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:28.337716    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db315d18-fe3b-4130-88db-118ffa42b534-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:13:29.021930 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:29.021895    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46" event={"ID":"db315d18-fe3b-4130-88db-118ffa42b534","Type":"ContainerDied","Data":"db384d64a3e65522d1f7db90c8c06d0995aa52302843baa21fb71417a3a53895"}
Jul 09 16:13:29.021930 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:29.021922    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflht46"
Jul 09 16:13:29.021930 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:29.021939    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db384d64a3e65522d1f7db90c8c06d0995aa52302843baa21fb71417a3a53895"
Jul 09 16:13:46.724368 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724334    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"]
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724577    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="util"
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724588    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="util"
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724598    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="pull"
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724603    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="pull"
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724609    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="extract"
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724614    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="extract"
Jul 09 16:13:46.724750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.724647    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="db315d18-fe3b-4130-88db-118ffa42b534" containerName="extract"
Jul 09 16:13:46.730000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.729983    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.734184 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.734164    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 16:13:46.734351 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.734327    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 16:13:46.734487 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.734459    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s4zxx\""
Jul 09 16:13:46.734487 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.734468    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 16:13:46.734598 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.734509    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 16:13:46.743551 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.743532    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"]
Jul 09 16:13:46.877801 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.877765    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-apiservice-cert\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.877801 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.877801    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7th9j\" (UniqueName: \"kubernetes.io/projected/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-kube-api-access-7th9j\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.878033 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.877836    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-manifests\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.878033 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.877866    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-tmp\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.878033 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.877910    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-webhook-cert\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.978361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.978253    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-manifests\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.978361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.978305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-tmp\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.978361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.978346    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-webhook-cert\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.978647 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.978385    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-apiservice-cert\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.978647 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.978403    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7th9j\" (UniqueName: \"kubernetes.io/projected/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-kube-api-access-7th9j\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.978755 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.978667    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-manifests\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.980771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.980743    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-tmp\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.980870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.980818    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-webhook-cert\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.980955 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.980919    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-apiservice-cert\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:46.988008 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:46.987987    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7th9j\" (UniqueName: \"kubernetes.io/projected/3fb1ebae-a1a3-468f-ada7-4f5ca58b7901-kube-api-access-7th9j\") pod \"opendatahub-operator-controller-manager-757d6df76f-cgmf8\" (UID: \"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901\") " pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:47.039988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:47.039954    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:47.182121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:47.182093    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"]
Jul 09 16:13:47.185478 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:13:47.185452    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb1ebae_a1a3_468f_ada7_4f5ca58b7901.slice/crio-d397b1d39d7288341d442ce7c3e448e4e921579a84b9feaf0225a8ed3a04eb11 WatchSource:0}: Error finding container d397b1d39d7288341d442ce7c3e448e4e921579a84b9feaf0225a8ed3a04eb11: Status 404 returned error can't find the container with id d397b1d39d7288341d442ce7c3e448e4e921579a84b9feaf0225a8ed3a04eb11
Jul 09 16:13:48.098418 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:48.098362    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8" event={"ID":"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901","Type":"ContainerStarted","Data":"d397b1d39d7288341d442ce7c3e448e4e921579a84b9feaf0225a8ed3a04eb11"}
Jul 09 16:13:51.113083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:51.113048    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8" event={"ID":"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901","Type":"ContainerStarted","Data":"4d1bc11bbb194c1b39600870db831ba7dc9d14897739afb4dafc700123f14b50"}
Jul 09 16:13:52.118086 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:52.118054    2558 generic.go:356] "Generic (PLEG): container finished" podID="3fb1ebae-a1a3-468f-ada7-4f5ca58b7901" containerID="4d1bc11bbb194c1b39600870db831ba7dc9d14897739afb4dafc700123f14b50" exitCode=0
Jul 09 16:13:52.118457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:52.118119    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8" event={"ID":"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901","Type":"ContainerDied","Data":"4d1bc11bbb194c1b39600870db831ba7dc9d14897739afb4dafc700123f14b50"}
Jul 09 16:13:53.124776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:53.124737    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8" event={"ID":"3fb1ebae-a1a3-468f-ada7-4f5ca58b7901","Type":"ContainerStarted","Data":"4830ae600a048d3e48f8b61b6f1a5ff1e1a01d2dd5aaea5508fda8a3fcc1a06f"}
Jul 09 16:13:53.125218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:53.124907    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:13:53.150237 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:13:53.150182    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8" podStartSLOduration=2.461198843 podStartE2EDuration="7.150165568s" totalImagesPullingTime="3.629908525s" totalInitContainerRuntime="1.0590582s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:13:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:13:53.148037709 +0000 UTC m=+578.529854498" watchObservedRunningTime="2026-07-09 16:13:53.150165568 +0000 UTC m=+578.531982354"
Jul 09 16:14:04.131684 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:04.131652    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-757d6df76f-cgmf8"
Jul 09 16:14:07.036284 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.036248    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"]
Jul 09 16:14:07.039742 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.039719    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.043022 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.043003    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 16:14:07.043117 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.043025    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 16:14:07.043329 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.043307    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 16:14:07.043574 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.043559    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 16:14:07.044828 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.044811    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4mtv2\""
Jul 09 16:14:07.053350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.053328    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"]
Jul 09 16:14:07.132429 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.132403    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfjdk\" (UniqueName: \"kubernetes.io/projected/12c930d5-0791-4b77-9875-eb7dd853ce00-kube-api-access-sfjdk\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.132569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.132455    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c930d5-0791-4b77-9875-eb7dd853ce00-tls-certs\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.132569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.132481    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12c930d5-0791-4b77-9875-eb7dd853ce00-tmp\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.233515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.233488    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c930d5-0791-4b77-9875-eb7dd853ce00-tls-certs\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.233515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.233517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12c930d5-0791-4b77-9875-eb7dd853ce00-tmp\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.233711 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.233557    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfjdk\" (UniqueName: \"kubernetes.io/projected/12c930d5-0791-4b77-9875-eb7dd853ce00-kube-api-access-sfjdk\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.233759 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:07.233703    2558 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 09 16:14:07.233839 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:07.233816    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12c930d5-0791-4b77-9875-eb7dd853ce00-tls-certs podName:12c930d5-0791-4b77-9875-eb7dd853ce00 nodeName:}" failed. No retries permitted until 2026-07-09 16:14:07.733791806 +0000 UTC m=+593.115608570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/12c930d5-0791-4b77-9875-eb7dd853ce00-tls-certs") pod "kube-auth-proxy-566b87dd6c-65ct9" (UID: "12c930d5-0791-4b77-9875-eb7dd853ce00") : secret "kube-auth-proxy-tls" not found
Jul 09 16:14:07.235974 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.235947    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/12c930d5-0791-4b77-9875-eb7dd853ce00-tmp\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.244985 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.244961    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfjdk\" (UniqueName: \"kubernetes.io/projected/12c930d5-0791-4b77-9875-eb7dd853ce00-kube-api-access-sfjdk\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.450463 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.450421    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c5466"]
Jul 09 16:14:07.454555 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.454528    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:07.457757 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.457729    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 16:14:07.458234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.458214    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-wv6cd\""
Jul 09 16:14:07.474032 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.473997    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c5466"]
Jul 09 16:14:07.637353 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.637309    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2smd9\" (UniqueName: \"kubernetes.io/projected/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-kube-api-access-2smd9\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:07.637539 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.637382    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-cert\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:07.738515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.738415    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c930d5-0791-4b77-9875-eb7dd853ce00-tls-certs\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.738515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.738479    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2smd9\" (UniqueName: \"kubernetes.io/projected/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-kube-api-access-2smd9\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:07.738515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.738510    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-cert\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:07.738802 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:07.738610    2558 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 16:14:07.738802 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:07.738668    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-cert podName:4d2bc1d3-113b-4e76-94a9-a162c915e7d0 nodeName:}" failed. No retries permitted until 2026-07-09 16:14:08.238652396 +0000 UTC m=+593.620469167 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-cert") pod "odh-model-controller-7998c67dc4-c5466" (UID: "4d2bc1d3-113b-4e76-94a9-a162c915e7d0") : secret "odh-model-controller-webhook-cert" not found
Jul 09 16:14:07.741087 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.741063    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12c930d5-0791-4b77-9875-eb7dd853ce00-tls-certs\") pod \"kube-auth-proxy-566b87dd6c-65ct9\" (UID: \"12c930d5-0791-4b77-9875-eb7dd853ce00\") " pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:07.759022 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.758990    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2smd9\" (UniqueName: \"kubernetes.io/projected/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-kube-api-access-2smd9\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:07.950897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:07.950843    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"
Jul 09 16:14:08.086406 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.086383    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9"]
Jul 09 16:14:08.088597 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:14:08.088549    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12c930d5_0791_4b77_9875_eb7dd853ce00.slice/crio-0adea27c07c53c3a5c0e62d5a82ff8c8f39b2114a3d26f4b6d4d973440de33ec WatchSource:0}: Error finding container 0adea27c07c53c3a5c0e62d5a82ff8c8f39b2114a3d26f4b6d4d973440de33ec: Status 404 returned error can't find the container with id 0adea27c07c53c3a5c0e62d5a82ff8c8f39b2114a3d26f4b6d4d973440de33ec
Jul 09 16:14:08.189746 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.189712    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9" event={"ID":"12c930d5-0791-4b77-9875-eb7dd853ce00","Type":"ContainerStarted","Data":"0adea27c07c53c3a5c0e62d5a82ff8c8f39b2114a3d26f4b6d4d973440de33ec"}
Jul 09 16:14:08.242241 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.242200    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-cert\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:08.244865 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.244838    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4d2bc1d3-113b-4e76-94a9-a162c915e7d0-cert\") pod \"odh-model-controller-7998c67dc4-c5466\" (UID: \"4d2bc1d3-113b-4e76-94a9-a162c915e7d0\") " pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:08.365162 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.365117    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:08.365821 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.365797    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"]
Jul 09 16:14:08.369623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.369607    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.373318 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.372627    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 16:14:08.373318 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.372666    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 16:14:08.373318 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.372668    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 16:14:08.373318 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.372632    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 16:14:08.373318 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.373147    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-nqqwh\""
Jul 09 16:14:08.386207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.386166    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"]
Jul 09 16:14:08.511602 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.511576    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-c5466"]
Jul 09 16:14:08.514372 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:14:08.514337    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2bc1d3_113b_4e76_94a9_a162c915e7d0.slice/crio-20af8c4f00770e92a3d3bbd78a3c1f13276e36a9401ab94ce473af0d87389bd0 WatchSource:0}: Error finding container 20af8c4f00770e92a3d3bbd78a3c1f13276e36a9401ab94ce473af0d87389bd0: Status 404 returned error can't find the container with id 20af8c4f00770e92a3d3bbd78a3c1f13276e36a9401ab94ce473af0d87389bd0
Jul 09 16:14:08.544449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544418    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-cacerts\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.544612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544461    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bcc739-3f8a-4699-8571-39d083aeeabb-local-certs\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.544612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544488    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.544612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrcb4\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-kube-api-access-nrcb4\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.544612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544554    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.544807 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.544807 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.544691    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-token\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.645990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.645898    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.645990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.645951    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.645990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.645988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-token\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.646242 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.646021    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-cacerts\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.646242 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.646052    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bcc739-3f8a-4699-8571-39d083aeeabb-local-certs\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.646410 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.646368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.646502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.646440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrcb4\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-kube-api-access-nrcb4\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.647328 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.647291    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.648658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.648628    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bcc739-3f8a-4699-8571-39d083aeeabb-local-certs\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.648658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.648651    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.648818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.648798    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-cacerts\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.649000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.648979    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.654255 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.654230    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-token\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.654481 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.654464    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrcb4\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-kube-api-access-nrcb4\") pod \"istiod-openshift-gateway-7574459656-xq4sj\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.696597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.696564    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:08.870164 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:08.870131    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"]
Jul 09 16:14:08.872052 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:14:08.872027    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7bcc739_3f8a_4699_8571_39d083aeeabb.slice/crio-b7f6f2bc6e0ae684ec960a58257342649325241fce84602630da95907fa1f7fe WatchSource:0}: Error finding container b7f6f2bc6e0ae684ec960a58257342649325241fce84602630da95907fa1f7fe: Status 404 returned error can't find the container with id b7f6f2bc6e0ae684ec960a58257342649325241fce84602630da95907fa1f7fe
Jul 09 16:14:09.196085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:09.196040    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c5466" event={"ID":"4d2bc1d3-113b-4e76-94a9-a162c915e7d0","Type":"ContainerStarted","Data":"20af8c4f00770e92a3d3bbd78a3c1f13276e36a9401ab94ce473af0d87389bd0"}
Jul 09 16:14:09.197506 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:09.197474    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" event={"ID":"f7bcc739-3f8a-4699-8571-39d083aeeabb","Type":"ContainerStarted","Data":"b7f6f2bc6e0ae684ec960a58257342649325241fce84602630da95907fa1f7fe"}
Jul 09 16:14:13.926747 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:13.926693    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:14:13.927229 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:13.926784    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:14:14.233759 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.233659    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9" event={"ID":"12c930d5-0791-4b77-9875-eb7dd853ce00","Type":"ContainerStarted","Data":"96404074ac9c3d39a007f1d73a77b90898dec07025662937c6e06705bfbc80fa"}
Jul 09 16:14:14.235186 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.235157    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-c5466" event={"ID":"4d2bc1d3-113b-4e76-94a9-a162c915e7d0","Type":"ContainerStarted","Data":"58d87f7b57367b1b4dd2ce0d8be00a89e509cf29255f81d0a29213810c3a4187"}
Jul 09 16:14:14.235312 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.235277    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:14.236688 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.236664    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" event={"ID":"f7bcc739-3f8a-4699-8571-39d083aeeabb","Type":"ContainerStarted","Data":"218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f"}
Jul 09 16:14:14.236818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.236725    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:14.253620 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.253578    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-566b87dd6c-65ct9" podStartSLOduration=1.414435031 podStartE2EDuration="7.253564763s" totalImagesPullingTime="5.839129732s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:14:14.252021366 +0000 UTC m=+599.633838153" watchObservedRunningTime="2026-07-09 16:14:14.253564763 +0000 UTC m=+599.635381550"
Jul 09 16:14:14.275569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.275508    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" podStartSLOduration=1.223339625 podStartE2EDuration="6.275488039s" totalImagesPullingTime="5.052148414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:14:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:14:14.274012653 +0000 UTC m=+599.655829439" watchObservedRunningTime="2026-07-09 16:14:14.275488039 +0000 UTC m=+599.657304830"
Jul 09 16:14:14.296566 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:14.296510    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-c5466" podStartSLOduration=1.8692902230000001 podStartE2EDuration="7.296493366s" totalImagesPullingTime="5.427203143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:14:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:14:14.294631398 +0000 UTC m=+599.676448184" watchObservedRunningTime="2026-07-09 16:14:14.296493366 +0000 UTC m=+599.678310152"
Jul 09 16:14:15.011151 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.011124    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:14:15.011550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.011214    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:14:15.242813 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.242779    2558 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-xq4sj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 16:14:15.242983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.242829    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" podUID="f7bcc739-3f8a-4699-8571-39d083aeeabb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:14:15.425103 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.425064    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"]
Jul 09 16:14:15.430004 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.429983    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.432990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.432970    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4fs7b\""
Jul 09 16:14:15.440590 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.440542    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"]
Jul 09 16:14:15.513013 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.512953    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513095    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513153    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpds6\" (UniqueName: \"kubernetes.io/projected/05d99eee-bd0f-41d3-8ef8-3104dc456277-kube-api-access-jpds6\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513198    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513227    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513256    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513298    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.513342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.513335    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d99eee-bd0f-41d3-8ef8-3104dc456277-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614454 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614421    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpds6\" (UniqueName: \"kubernetes.io/projected/05d99eee-bd0f-41d3-8ef8-3104dc456277-kube-api-access-jpds6\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614467    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614487    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614503    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614529    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614842 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614660    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d99eee-bd0f-41d3-8ef8-3104dc456277-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614842 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614715    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614842 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614774    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.614842 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614799    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.615059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.614957    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.615059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.615029    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.615613 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.615587    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.615896 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.615657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.615896 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.615779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05d99eee-bd0f-41d3-8ef8-3104dc456277-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.616966 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.616940    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.617519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.617501    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.628914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.628868    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpds6\" (UniqueName: \"kubernetes.io/projected/05d99eee-bd0f-41d3-8ef8-3104dc456277-kube-api-access-jpds6\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.629000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.628966    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05d99eee-bd0f-41d3-8ef8-3104dc456277-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb6788znpb\" (UID: \"05d99eee-bd0f-41d3-8ef8-3104dc456277\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.740605 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.740524    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:15.895511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:15.895485    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"]
Jul 09 16:14:15.897692 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:14:15.897665    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d99eee_bd0f_41d3_8ef8_3104dc456277.slice/crio-52317e085fdd647d11535dad813a0de512923384732e14d2afa9599bcf34fa23 WatchSource:0}: Error finding container 52317e085fdd647d11535dad813a0de512923384732e14d2afa9599bcf34fa23: Status 404 returned error can't find the container with id 52317e085fdd647d11535dad813a0de512923384732e14d2afa9599bcf34fa23
Jul 09 16:14:16.245299 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:16.245269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb" event={"ID":"05d99eee-bd0f-41d3-8ef8-3104dc456277","Type":"ContainerStarted","Data":"52317e085fdd647d11535dad813a0de512923384732e14d2afa9599bcf34fa23"}
Jul 09 16:14:17.919557 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.919518    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"]
Jul 09 16:14:17.927779 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.927062    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:17.927779 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.927098    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"]
Jul 09 16:14:17.930622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.930329    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:17.932432 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.931204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-rbxhj\""
Jul 09 16:14:17.932432 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.931427    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 09 16:14:17.935775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.935292    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"]
Jul 09 16:14:17.935775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.935688    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 09 16:14:17.935775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.935772    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-jvjxq\""
Jul 09 16:14:17.944112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:17.943296    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"]
Jul 09 16:14:18.036936 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.036871    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9sf7\" (UID: \"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.037115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.036956    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w84cr\" (UniqueName: \"kubernetes.io/projected/da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8-kube-api-access-w84cr\") pod \"llmisvc-controller-manager-777c6bd7d5-b9sf7\" (UID: \"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.037115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.037041    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm42\" (UniqueName: \"kubernetes.io/projected/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-kube-api-access-thm42\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.037115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.037088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.138463 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.138426    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9sf7\" (UID: \"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.138652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.138486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w84cr\" (UniqueName: \"kubernetes.io/projected/da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8-kube-api-access-w84cr\") pod \"llmisvc-controller-manager-777c6bd7d5-b9sf7\" (UID: \"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.138652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.138528    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm42\" (UniqueName: \"kubernetes.io/projected/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-kube-api-access-thm42\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.138652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.138565    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.138799 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:18.138723    2558 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 09 16:14:18.139012 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:18.138977    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-cert podName:0fec7e61-b970-440e-aa5c-0bfcf20f8b3e nodeName:}" failed. No retries permitted until 2026-07-09 16:14:18.638859755 +0000 UTC m=+604.020676543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-cert") pod "kserve-controller-manager-d6f4b65b8-zpfzl" (UID: "0fec7e61-b970-440e-aa5c-0bfcf20f8b3e") : secret "kserve-webhook-server-cert" not found
Jul 09 16:14:18.141556 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.141535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-b9sf7\" (UID: \"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.156732 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.156706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w84cr\" (UniqueName: \"kubernetes.io/projected/da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8-kube-api-access-w84cr\") pod \"llmisvc-controller-manager-777c6bd7d5-b9sf7\" (UID: \"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.157255 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.157233    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm42\" (UniqueName: \"kubernetes.io/projected/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-kube-api-access-thm42\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.242513 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.242424    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:18.254735 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.254712    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:18.599630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.599581    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:14:18.599734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.599666    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:14:18.599734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.599713    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:14:18.643841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.643553    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.646269 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.646245    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0fec7e61-b970-440e-aa5c-0bfcf20f8b3e-cert\") pod \"kserve-controller-manager-d6f4b65b8-zpfzl\" (UID: \"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.677507 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.677480    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"]
Jul 09 16:14:18.678505 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:14:18.678477    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podda7d7c25_f7f2_4df0_b5d2_d01dcb40ead8.slice/crio-f86e97f862f7cf2cc1e20d5ae0e747d3f3afcbc27d4cdeda2c83f86a7246ad2d WatchSource:0}: Error finding container f86e97f862f7cf2cc1e20d5ae0e747d3f3afcbc27d4cdeda2c83f86a7246ad2d: Status 404 returned error can't find the container with id f86e97f862f7cf2cc1e20d5ae0e747d3f3afcbc27d4cdeda2c83f86a7246ad2d
Jul 09 16:14:18.848319 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.848290    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:18.995698 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:18.995653    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"]
Jul 09 16:14:18.998258 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:14:18.998228    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fec7e61_b970_440e_aa5c_0bfcf20f8b3e.slice/crio-c61ba93f3e4e49e84e9b7c79921af606e1bd655c4ddadb0f2a13af2b3651eaa4 WatchSource:0}: Error finding container c61ba93f3e4e49e84e9b7c79921af606e1bd655c4ddadb0f2a13af2b3651eaa4: Status 404 returned error can't find the container with id c61ba93f3e4e49e84e9b7c79921af606e1bd655c4ddadb0f2a13af2b3651eaa4
Jul 09 16:14:19.262978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:19.262892    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7" event={"ID":"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8","Type":"ContainerStarted","Data":"f86e97f862f7cf2cc1e20d5ae0e747d3f3afcbc27d4cdeda2c83f86a7246ad2d"}
Jul 09 16:14:19.264402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:19.264375    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb" event={"ID":"05d99eee-bd0f-41d3-8ef8-3104dc456277","Type":"ContainerStarted","Data":"9266679799c77f5b5baaa9cce7e4bab8702a29d8b88ead51bece2ffe4304d14f"}
Jul 09 16:14:19.265440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:19.265415    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl" event={"ID":"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e","Type":"ContainerStarted","Data":"c61ba93f3e4e49e84e9b7c79921af606e1bd655c4ddadb0f2a13af2b3651eaa4"}
Jul 09 16:14:19.303672 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:19.303621    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb" podStartSLOduration=1.6039208550000001 podStartE2EDuration="4.303603486s" totalImagesPullingTime="2.699682631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:14:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:14:19.302408906 +0000 UTC m=+604.684225693" watchObservedRunningTime="2026-07-09 16:14:19.303603486 +0000 UTC m=+604.685420272"
Jul 09 16:14:19.740859 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:19.740819    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:19.746392 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:19.746342    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:20.271125 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.271085    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:20.272217 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.272194    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb6788znpb"
Jul 09 16:14:20.536261 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.536178    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"]
Jul 09 16:14:20.536397 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.536367    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" podUID="f7bcc739-3f8a-4699-8571-39d083aeeabb" containerName="discovery" containerID="cri-o://218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f" gracePeriod=30
Jul 09 16:14:20.783862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.783838    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:20.874666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.874636    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-kubeconfig\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.874850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.874815    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-ca-configmap\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.874954 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.874850    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-cacerts\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.874954 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.874894    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bcc739-3f8a-4699-8571-39d083aeeabb-local-certs\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.874954 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.874911    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrcb4\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-kube-api-access-nrcb4\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.875127 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.875100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-dns-cert\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.875181 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.875159    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-token\") pod \"f7bcc739-3f8a-4699-8571-39d083aeeabb\" (UID: \"f7bcc739-3f8a-4699-8571-39d083aeeabb\") "
Jul 09 16:14:20.875856 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.875802    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:14:20.877800 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.877773    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:14:20.878000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.877922    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-cacerts" (OuterVolumeSpecName: "cacerts") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:14:20.878000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.877935    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7bcc739-3f8a-4699-8571-39d083aeeabb-local-certs" (OuterVolumeSpecName: "local-certs") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:14:20.878132 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.878017    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:14:20.878312 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.878279    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-kube-api-access-nrcb4" (OuterVolumeSpecName: "kube-api-access-nrcb4") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "kube-api-access-nrcb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:14:20.878375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.878357    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f7bcc739-3f8a-4699-8571-39d083aeeabb" (UID: "f7bcc739-3f8a-4699-8571-39d083aeeabb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:14:20.976491 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976455    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-ca-configmap\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:20.976491 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976485    2558 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-cacerts\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:20.976491 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976498    2558 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7bcc739-3f8a-4699-8571-39d083aeeabb-local-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:20.976692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976507    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrcb4\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-kube-api-access-nrcb4\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:20.976692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976516    2558 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-csr-dns-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:20.976692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976524    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:20.976692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:20.976532    2558 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f7bcc739-3f8a-4699-8571-39d083aeeabb-istio-kubeconfig\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:14:21.276023 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.275985    2558 generic.go:356] "Generic (PLEG): container finished" podID="f7bcc739-3f8a-4699-8571-39d083aeeabb" containerID="218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f" exitCode=0
Jul 09 16:14:21.276488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.276040    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"
Jul 09 16:14:21.276488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.276076    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" event={"ID":"f7bcc739-3f8a-4699-8571-39d083aeeabb","Type":"ContainerDied","Data":"218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f"}
Jul 09 16:14:21.276488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.276142    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj" event={"ID":"f7bcc739-3f8a-4699-8571-39d083aeeabb","Type":"ContainerDied","Data":"b7f6f2bc6e0ae684ec960a58257342649325241fce84602630da95907fa1f7fe"}
Jul 09 16:14:21.276488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.276191    2558 scope.go:117] "RemoveContainer" containerID="218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f"
Jul 09 16:14:21.284666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.284648    2558 scope.go:117] "RemoveContainer" containerID="218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f"
Jul 09 16:14:21.284947 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:14:21.284921    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f\": container with ID starting with 218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f not found: ID does not exist" containerID="218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f"
Jul 09 16:14:21.284995 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.284959    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f"} err="failed to get container status \"218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f\": rpc error: code = NotFound desc = could not find container \"218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f\": container with ID starting with 218ba1fcdd8cadd9576081de6dc63b12c5c6f08c68e10a23be59c6ae8d7a7c7f not found: ID does not exist"
Jul 09 16:14:21.295826 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.295783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"]
Jul 09 16:14:21.299786 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:21.299764    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-xq4sj"]
Jul 09 16:14:23.178026 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.177984    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7bcc739-3f8a-4699-8571-39d083aeeabb" path="/var/lib/kubelet/pods/f7bcc739-3f8a-4699-8571-39d083aeeabb/volumes"
Jul 09 16:14:23.288254 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.288218    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7" event={"ID":"da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8","Type":"ContainerStarted","Data":"fb3a4da66dea12a575317aa7b8126b8789a44ab011e5184ebcb46b7988e1868b"}
Jul 09 16:14:23.288452 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.288380    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:23.289695 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.289676    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl" event={"ID":"0fec7e61-b970-440e-aa5c-0bfcf20f8b3e","Type":"ContainerStarted","Data":"f4ee4badbcdf3b481daec0f4f3c731208d7ef3eae7400c7a8c6d2ebe6e3be979"}
Jul 09 16:14:23.289818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.289738    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:14:23.305401 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.305352    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7" podStartSLOduration=2.412576336 podStartE2EDuration="6.305338833s" totalImagesPullingTime="3.892762497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:14:23.304519599 +0000 UTC m=+608.686336386" watchObservedRunningTime="2026-07-09 16:14:23.305338833 +0000 UTC m=+608.687155619"
Jul 09 16:14:23.321575 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:23.321523    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl" podStartSLOduration=2.745997827 podStartE2EDuration="6.321504168s" totalImagesPullingTime="3.575506341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:14:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:14:23.319813856 +0000 UTC m=+608.701630643" watchObservedRunningTime="2026-07-09 16:14:23.321504168 +0000 UTC m=+608.703320956"
Jul 09 16:14:25.243629 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:25.243600    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-c5466"
Jul 09 16:14:54.298871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:54.298837    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-b9sf7"
Jul 09 16:14:54.301802 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:14:54.301777    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-zpfzl"
Jul 09 16:15:08.214587 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.214555    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zdgjd"]
Jul 09 16:15:08.215180 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.215006    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7bcc739-3f8a-4699-8571-39d083aeeabb" containerName="discovery"
Jul 09 16:15:08.215180 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.215026    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7bcc739-3f8a-4699-8571-39d083aeeabb" containerName="discovery"
Jul 09 16:15:08.215180 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.215108    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7bcc739-3f8a-4699-8571-39d083aeeabb" containerName="discovery"
Jul 09 16:15:08.217106 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.217084    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:08.220157 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.220129    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 16:15:08.220279 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.220161    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-p2z26\""
Jul 09 16:15:08.220279 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.220130    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 16:15:08.229308 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.229285    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zdgjd"]
Jul 09 16:15:08.258582 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.258551    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrnpb\" (UniqueName: \"kubernetes.io/projected/a8f28f7a-ce64-46e3-b366-4961e13988ff-kube-api-access-mrnpb\") pod \"kuadrant-operator-catalog-zdgjd\" (UID: \"a8f28f7a-ce64-46e3-b366-4961e13988ff\") " pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:08.360064 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.360028    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrnpb\" (UniqueName: \"kubernetes.io/projected/a8f28f7a-ce64-46e3-b366-4961e13988ff-kube-api-access-mrnpb\") pod \"kuadrant-operator-catalog-zdgjd\" (UID: \"a8f28f7a-ce64-46e3-b366-4961e13988ff\") " pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:08.368712 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.368689    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrnpb\" (UniqueName: \"kubernetes.io/projected/a8f28f7a-ce64-46e3-b366-4961e13988ff-kube-api-access-mrnpb\") pod \"kuadrant-operator-catalog-zdgjd\" (UID: \"a8f28f7a-ce64-46e3-b366-4961e13988ff\") " pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:08.526783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.526677    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:08.580846 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.580816    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zdgjd"]
Jul 09 16:15:08.656988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.656965    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zdgjd"]
Jul 09 16:15:08.659298 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:08.659271    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f28f7a_ce64_46e3_b366_4961e13988ff.slice/crio-a7d69dbb34c1d35495d1b87bb3cc4d751a0b6d301e01e50c80a3fa6040430489 WatchSource:0}: Error finding container a7d69dbb34c1d35495d1b87bb3cc4d751a0b6d301e01e50c80a3fa6040430489: Status 404 returned error can't find the container with id a7d69dbb34c1d35495d1b87bb3cc4d751a0b6d301e01e50c80a3fa6040430489
Jul 09 16:15:08.790984 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.790898    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k8xfw"]
Jul 09 16:15:08.797043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.797013    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:08.801847 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.801809    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k8xfw"]
Jul 09 16:15:08.864053 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.864011    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv94w\" (UniqueName: \"kubernetes.io/projected/567f9a6c-5db1-4a88-b251-cd767e7bdb4b-kube-api-access-mv94w\") pod \"kuadrant-operator-catalog-k8xfw\" (UID: \"567f9a6c-5db1-4a88-b251-cd767e7bdb4b\") " pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:08.964991 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.964953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv94w\" (UniqueName: \"kubernetes.io/projected/567f9a6c-5db1-4a88-b251-cd767e7bdb4b-kube-api-access-mv94w\") pod \"kuadrant-operator-catalog-k8xfw\" (UID: \"567f9a6c-5db1-4a88-b251-cd767e7bdb4b\") " pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:08.973351 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:08.973323    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv94w\" (UniqueName: \"kubernetes.io/projected/567f9a6c-5db1-4a88-b251-cd767e7bdb4b-kube-api-access-mv94w\") pod \"kuadrant-operator-catalog-k8xfw\" (UID: \"567f9a6c-5db1-4a88-b251-cd767e7bdb4b\") " pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:09.106817 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:09.106781    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:09.247938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:09.247913    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k8xfw"]
Jul 09 16:15:09.249860 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:09.249835    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod567f9a6c_5db1_4a88_b251_cd767e7bdb4b.slice/crio-cc6ca06d2e1b64a3fc06c36e5b5c1d052d8e1fe67f9cfdb9da2f882a5ed3b25a WatchSource:0}: Error finding container cc6ca06d2e1b64a3fc06c36e5b5c1d052d8e1fe67f9cfdb9da2f882a5ed3b25a: Status 404 returned error can't find the container with id cc6ca06d2e1b64a3fc06c36e5b5c1d052d8e1fe67f9cfdb9da2f882a5ed3b25a
Jul 09 16:15:09.492228 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:09.492147    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw" event={"ID":"567f9a6c-5db1-4a88-b251-cd767e7bdb4b","Type":"ContainerStarted","Data":"cc6ca06d2e1b64a3fc06c36e5b5c1d052d8e1fe67f9cfdb9da2f882a5ed3b25a"}
Jul 09 16:15:09.493299 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:09.493275    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd" event={"ID":"a8f28f7a-ce64-46e3-b366-4961e13988ff","Type":"ContainerStarted","Data":"a7d69dbb34c1d35495d1b87bb3cc4d751a0b6d301e01e50c80a3fa6040430489"}
Jul 09 16:15:11.507296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.507251    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd" event={"ID":"a8f28f7a-ce64-46e3-b366-4961e13988ff","Type":"ContainerStarted","Data":"1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e"}
Jul 09 16:15:11.507731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.507294    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd" podUID="a8f28f7a-ce64-46e3-b366-4961e13988ff" containerName="registry-server" containerID="cri-o://1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e" gracePeriod=2
Jul 09 16:15:11.508788 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.508754    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw" event={"ID":"567f9a6c-5db1-4a88-b251-cd767e7bdb4b","Type":"ContainerStarted","Data":"29f6dd15ad9335c1042e48fe6c54120fc8581623a2894b7c28f7de0b75eb2783"}
Jul 09 16:15:11.523428 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.523390    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd" podStartSLOduration=1.345024378 podStartE2EDuration="3.523376159s" totalImagesPullingTime="2.178351781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:11.522597836 +0000 UTC m=+656.904414623" watchObservedRunningTime="2026-07-09 16:15:11.523376159 +0000 UTC m=+656.905192961"
Jul 09 16:15:11.537289 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.537237    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw" podStartSLOduration=1.9531701 podStartE2EDuration="3.537220503s" totalImagesPullingTime="1.584050403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:11.537212081 +0000 UTC m=+656.919028872" watchObservedRunningTime="2026-07-09 16:15:11.537220503 +0000 UTC m=+656.919037292"
Jul 09 16:15:11.749023 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.748999    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:11.789547 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.789486    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrnpb\" (UniqueName: \"kubernetes.io/projected/a8f28f7a-ce64-46e3-b366-4961e13988ff-kube-api-access-mrnpb\") pod \"a8f28f7a-ce64-46e3-b366-4961e13988ff\" (UID: \"a8f28f7a-ce64-46e3-b366-4961e13988ff\") "
Jul 09 16:15:11.791619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.791598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f28f7a-ce64-46e3-b366-4961e13988ff-kube-api-access-mrnpb" (OuterVolumeSpecName: "kube-api-access-mrnpb") pod "a8f28f7a-ce64-46e3-b366-4961e13988ff" (UID: "a8f28f7a-ce64-46e3-b366-4961e13988ff"). InnerVolumeSpecName "kube-api-access-mrnpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:15:11.890720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:11.890691    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrnpb\" (UniqueName: \"kubernetes.io/projected/a8f28f7a-ce64-46e3-b366-4961e13988ff-kube-api-access-mrnpb\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:12.514333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.514295    2558 generic.go:356] "Generic (PLEG): container finished" podID="a8f28f7a-ce64-46e3-b366-4961e13988ff" containerID="1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e" exitCode=0
Jul 09 16:15:12.514818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.514341    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd"
Jul 09 16:15:12.514818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.514394    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd" event={"ID":"a8f28f7a-ce64-46e3-b366-4961e13988ff","Type":"ContainerDied","Data":"1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e"}
Jul 09 16:15:12.514818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.514453    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zdgjd" event={"ID":"a8f28f7a-ce64-46e3-b366-4961e13988ff","Type":"ContainerDied","Data":"a7d69dbb34c1d35495d1b87bb3cc4d751a0b6d301e01e50c80a3fa6040430489"}
Jul 09 16:15:12.514818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.514485    2558 scope.go:117] "RemoveContainer" containerID="1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e"
Jul 09 16:15:12.524125 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.524106    2558 scope.go:117] "RemoveContainer" containerID="1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e"
Jul 09 16:15:12.524387 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:15:12.524369    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e\": container with ID starting with 1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e not found: ID does not exist" containerID="1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e"
Jul 09 16:15:12.524435 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.524396    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e"} err="failed to get container status \"1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e\": rpc error: code = NotFound desc = could not find container \"1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e\": container with ID starting with 1d7471774831320671d020ab2e8d684d7d3b108671638ccaf40fc7b58dc9e16e not found: ID does not exist"
Jul 09 16:15:12.536559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.536526    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zdgjd"]
Jul 09 16:15:12.540389 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:12.540364    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zdgjd"]
Jul 09 16:15:13.177002 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:13.176973    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8f28f7a-ce64-46e3-b366-4961e13988ff" path="/var/lib/kubelet/pods/a8f28f7a-ce64-46e3-b366-4961e13988ff/volumes"
Jul 09 16:15:19.107017 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:19.106976    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:19.107405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:19.107047    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:19.130100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:19.130070    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:19.568115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:19.568088    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-k8xfw"
Jul 09 16:15:24.025271 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.025236    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"]
Jul 09 16:15:24.025638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.025515    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8f28f7a-ce64-46e3-b366-4961e13988ff" containerName="registry-server"
Jul 09 16:15:24.025638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.025525    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8f28f7a-ce64-46e3-b366-4961e13988ff" containerName="registry-server"
Jul 09 16:15:24.025638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.025582    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8f28f7a-ce64-46e3-b366-4961e13988ff" containerName="registry-server"
Jul 09 16:15:24.028714 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.028697    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.031512 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.031494    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-b82xm\""
Jul 09 16:15:24.035744 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.035717    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"]
Jul 09 16:15:24.091711 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.091673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.091925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.091758    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flwwq\" (UniqueName: \"kubernetes.io/projected/21ff865d-6fc8-4ec9-a663-5a6728b6c993-kube-api-access-flwwq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.091925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.091792    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.192775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.192738    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.192993 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.192822    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flwwq\" (UniqueName: \"kubernetes.io/projected/21ff865d-6fc8-4ec9-a663-5a6728b6c993-kube-api-access-flwwq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.192993 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.192845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.193208 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.193182    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.193208 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.193200    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.202904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.202866    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flwwq\" (UniqueName: \"kubernetes.io/projected/21ff865d-6fc8-4ec9-a663-5a6728b6c993-kube-api-access-flwwq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.338506 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.338476    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:24.488288 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.488164    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"]
Jul 09 16:15:24.490375 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:24.490350    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ff865d_6fc8_4ec9_a663_5a6728b6c993.slice/crio-096be4748468113e85ecabfce71a2cf71b6910e162492f3619da03ac0de6c5be WatchSource:0}: Error finding container 096be4748468113e85ecabfce71a2cf71b6910e162492f3619da03ac0de6c5be: Status 404 returned error can't find the container with id 096be4748468113e85ecabfce71a2cf71b6910e162492f3619da03ac0de6c5be
Jul 09 16:15:24.568186 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.568148    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9" event={"ID":"21ff865d-6fc8-4ec9-a663-5a6728b6c993","Type":"ContainerStarted","Data":"0dbd1cc797db4328f43ce255cd7a53c3dfc7e689b5ff86297a6cc5a849255128"}
Jul 09 16:15:24.568358 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.568201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9" event={"ID":"21ff865d-6fc8-4ec9-a663-5a6728b6c993","Type":"ContainerStarted","Data":"096be4748468113e85ecabfce71a2cf71b6910e162492f3619da03ac0de6c5be"}
Jul 09 16:15:24.824924 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.824888    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"]
Jul 09 16:15:24.827328 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.827309    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:24.836152 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.836130    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"]
Jul 09 16:15:24.899391 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.899358    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwr6g\" (UniqueName: \"kubernetes.io/projected/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-kube-api-access-qwr6g\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:24.899391 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.899406    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:24.899621 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:24.899429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.000200 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.000154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwr6g\" (UniqueName: \"kubernetes.io/projected/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-kube-api-access-qwr6g\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.000377 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.000229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.000377 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.000261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.000638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.000617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.000712 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.000652    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.009338 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.009304    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwr6g\" (UniqueName: \"kubernetes.io/projected/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-kube-api-access-qwr6g\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.136631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.136537    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:25.269153 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.269131    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"]
Jul 09 16:15:25.271256 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:25.271228    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb63f40_aeb9_487b_8bb2_e2ba3977ae8b.slice/crio-01e092bee1a67cde52486cb3288fe0be2e86575fa0487c5e31a14f40e3214744 WatchSource:0}: Error finding container 01e092bee1a67cde52486cb3288fe0be2e86575fa0487c5e31a14f40e3214744: Status 404 returned error can't find the container with id 01e092bee1a67cde52486cb3288fe0be2e86575fa0487c5e31a14f40e3214744
Jul 09 16:15:25.419232 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.419155    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"]
Jul 09 16:15:25.421785 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.421768    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.429983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.429959    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"]
Jul 09 16:15:25.503679 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.503645    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.503679 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.503688    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjgpv\" (UniqueName: \"kubernetes.io/projected/418616e9-35f0-4127-b3f3-9c3d582ff28c-kube-api-access-rjgpv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.503918 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.503718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.573767 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.573733    2558 generic.go:356] "Generic (PLEG): container finished" podID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerID="7923334790af43ec6acab77617491e648d7512b6deaf58e97f15bb805f5138d7" exitCode=0
Jul 09 16:15:25.573938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.573820    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh" event={"ID":"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b","Type":"ContainerDied","Data":"7923334790af43ec6acab77617491e648d7512b6deaf58e97f15bb805f5138d7"}
Jul 09 16:15:25.573938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.573914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh" event={"ID":"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b","Type":"ContainerStarted","Data":"01e092bee1a67cde52486cb3288fe0be2e86575fa0487c5e31a14f40e3214744"}
Jul 09 16:15:25.575420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.575399    2558 generic.go:356] "Generic (PLEG): container finished" podID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerID="0dbd1cc797db4328f43ce255cd7a53c3dfc7e689b5ff86297a6cc5a849255128" exitCode=0
Jul 09 16:15:25.575507 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.575470    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9" event={"ID":"21ff865d-6fc8-4ec9-a663-5a6728b6c993","Type":"ContainerDied","Data":"0dbd1cc797db4328f43ce255cd7a53c3dfc7e689b5ff86297a6cc5a849255128"}
Jul 09 16:15:25.604616 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.604587    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.604805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.604692    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.604805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.604722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjgpv\" (UniqueName: \"kubernetes.io/projected/418616e9-35f0-4127-b3f3-9c3d582ff28c-kube-api-access-rjgpv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.605020 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.604997    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.605079 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.605006    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.614661 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.614640    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjgpv\" (UniqueName: \"kubernetes.io/projected/418616e9-35f0-4127-b3f3-9c3d582ff28c-kube-api-access-rjgpv\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.751834 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.751734    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:25.828901 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.827108    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"]
Jul 09 16:15:25.834783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.834755    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:25.836125 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.836096    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"]
Jul 09 16:15:25.902235 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.902208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"]
Jul 09 16:15:25.903865 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:25.903835    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418616e9_35f0_4127_b3f3_9c3d582ff28c.slice/crio-2e6519f66f5632ea30ba290024ed8d0da0638a32752e1a1bda86dcf96d207de1 WatchSource:0}: Error finding container 2e6519f66f5632ea30ba290024ed8d0da0638a32752e1a1bda86dcf96d207de1: Status 404 returned error can't find the container with id 2e6519f66f5632ea30ba290024ed8d0da0638a32752e1a1bda86dcf96d207de1
Jul 09 16:15:25.907730 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.907704    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgngv\" (UniqueName: \"kubernetes.io/projected/c34ab2f4-8edc-4723-8578-3edfdb0e7169-kube-api-access-jgngv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:25.907816 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.907747    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:25.907816 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:25.907779    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.009033 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.009008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgngv\" (UniqueName: \"kubernetes.io/projected/c34ab2f4-8edc-4723-8578-3edfdb0e7169-kube-api-access-jgngv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.009123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.009056    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.009123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.009083    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.009493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.009435    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.009577 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.009521    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.018157 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.018134    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgngv\" (UniqueName: \"kubernetes.io/projected/c34ab2f4-8edc-4723-8578-3edfdb0e7169-kube-api-access-jgngv\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.146241 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.146196    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:26.287561 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.287536    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"]
Jul 09 16:15:26.292159 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:26.292130    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc34ab2f4_8edc_4723_8578_3edfdb0e7169.slice/crio-c6bb59c7e19e2274eb65c0c27f3fde07c514509efb978e8aae63080e0f05fbcf WatchSource:0}: Error finding container c6bb59c7e19e2274eb65c0c27f3fde07c514509efb978e8aae63080e0f05fbcf: Status 404 returned error can't find the container with id c6bb59c7e19e2274eb65c0c27f3fde07c514509efb978e8aae63080e0f05fbcf
Jul 09 16:15:26.582188 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.582155    2558 generic.go:356] "Generic (PLEG): container finished" podID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerID="7e8adf7d25154cc5ac1c236736163dd34e27da235c7125b2d26c313b29f0bf89" exitCode=0
Jul 09 16:15:26.582362 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.582249    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv" event={"ID":"c34ab2f4-8edc-4723-8578-3edfdb0e7169","Type":"ContainerDied","Data":"7e8adf7d25154cc5ac1c236736163dd34e27da235c7125b2d26c313b29f0bf89"}
Jul 09 16:15:26.582362 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.582297    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv" event={"ID":"c34ab2f4-8edc-4723-8578-3edfdb0e7169","Type":"ContainerStarted","Data":"c6bb59c7e19e2274eb65c0c27f3fde07c514509efb978e8aae63080e0f05fbcf"}
Jul 09 16:15:26.584007 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.583984    2558 generic.go:356] "Generic (PLEG): container finished" podID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerID="e39bb58bbb7a9ba07a8b61a8a76735be5622289d4206a56f18d50be11e8d2f48" exitCode=0
Jul 09 16:15:26.584124 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.584084    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4" event={"ID":"418616e9-35f0-4127-b3f3-9c3d582ff28c","Type":"ContainerDied","Data":"e39bb58bbb7a9ba07a8b61a8a76735be5622289d4206a56f18d50be11e8d2f48"}
Jul 09 16:15:26.584186 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.584136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4" event={"ID":"418616e9-35f0-4127-b3f3-9c3d582ff28c","Type":"ContainerStarted","Data":"2e6519f66f5632ea30ba290024ed8d0da0638a32752e1a1bda86dcf96d207de1"}
Jul 09 16:15:26.586211 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.586188    2558 generic.go:356] "Generic (PLEG): container finished" podID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerID="5ac31e11688afd29f27adea2f669738d9d6f54d69fb7c86476d1c82fb2a37ac0" exitCode=0
Jul 09 16:15:26.586317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:26.586233    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh" event={"ID":"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b","Type":"ContainerDied","Data":"5ac31e11688afd29f27adea2f669738d9d6f54d69fb7c86476d1c82fb2a37ac0"}
Jul 09 16:15:27.592763 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:27.592731    2558 generic.go:356] "Generic (PLEG): container finished" podID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerID="ee3c3c8e78462f7beacf0acfdd336779d21784edbe55a19a26dbcf1a86e63016" exitCode=0
Jul 09 16:15:27.593214 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:27.592818    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9" event={"ID":"21ff865d-6fc8-4ec9-a663-5a6728b6c993","Type":"ContainerDied","Data":"ee3c3c8e78462f7beacf0acfdd336779d21784edbe55a19a26dbcf1a86e63016"}
Jul 09 16:15:27.595083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:27.595061    2558 generic.go:356] "Generic (PLEG): container finished" podID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerID="90ea9a3089e6ddd458865f44a1971ea6d45a3cf154e18a2536a41e4587ccb217" exitCode=0
Jul 09 16:15:27.595180 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:27.595136    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh" event={"ID":"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b","Type":"ContainerDied","Data":"90ea9a3089e6ddd458865f44a1971ea6d45a3cf154e18a2536a41e4587ccb217"}
Jul 09 16:15:27.596987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:27.596963    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4" event={"ID":"418616e9-35f0-4127-b3f3-9c3d582ff28c","Type":"ContainerStarted","Data":"6332f5ea35c02902395a998ce9fabf096e70a08465bb2417c3c587e58a3f3012"}
Jul 09 16:15:28.603543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.603504    2558 generic.go:356] "Generic (PLEG): container finished" podID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerID="1286012a0fc691a97686c8567780b2e1fd54160e5d4570dc25f1d75fa43e9c3a" exitCode=0
Jul 09 16:15:28.603950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.603565    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv" event={"ID":"c34ab2f4-8edc-4723-8578-3edfdb0e7169","Type":"ContainerDied","Data":"1286012a0fc691a97686c8567780b2e1fd54160e5d4570dc25f1d75fa43e9c3a"}
Jul 09 16:15:28.605858 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.605833    2558 generic.go:356] "Generic (PLEG): container finished" podID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerID="6332f5ea35c02902395a998ce9fabf096e70a08465bb2417c3c587e58a3f3012" exitCode=0
Jul 09 16:15:28.606046 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.605914    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4" event={"ID":"418616e9-35f0-4127-b3f3-9c3d582ff28c","Type":"ContainerDied","Data":"6332f5ea35c02902395a998ce9fabf096e70a08465bb2417c3c587e58a3f3012"}
Jul 09 16:15:28.608266 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.608235    2558 generic.go:356] "Generic (PLEG): container finished" podID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerID="04791d2eb052116fe12352c246835101e969c8f1c3e9cb76c1a5e7cdfbf2c51e" exitCode=0
Jul 09 16:15:28.608350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.608320    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9" event={"ID":"21ff865d-6fc8-4ec9-a663-5a6728b6c993","Type":"ContainerDied","Data":"04791d2eb052116fe12352c246835101e969c8f1c3e9cb76c1a5e7cdfbf2c51e"}
Jul 09 16:15:28.757603 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.757577    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:28.835279 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.835253    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwr6g\" (UniqueName: \"kubernetes.io/projected/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-kube-api-access-qwr6g\") pod \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") "
Jul 09 16:15:28.835375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.835288    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-util\") pod \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") "
Jul 09 16:15:28.835375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.835333    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-bundle\") pod \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\" (UID: \"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b\") "
Jul 09 16:15:28.836078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.836048    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-bundle" (OuterVolumeSpecName: "bundle") pod "3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" (UID: "3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:28.837367 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.837344    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-kube-api-access-qwr6g" (OuterVolumeSpecName: "kube-api-access-qwr6g") pod "3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" (UID: "3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b"). InnerVolumeSpecName "kube-api-access-qwr6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:15:28.840674 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.840655    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-util" (OuterVolumeSpecName: "util") pod "3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" (UID: "3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:28.936233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.936174    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:28.936233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.936198    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:28.936233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:28.936207    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwr6g\" (UniqueName: \"kubernetes.io/projected/3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b-kube-api-access-qwr6g\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:29.614739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.614705    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh"
Jul 09 16:15:29.614739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.614718    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh" event={"ID":"3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b","Type":"ContainerDied","Data":"01e092bee1a67cde52486cb3288fe0be2e86575fa0487c5e31a14f40e3214744"}
Jul 09 16:15:29.615202 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.614760    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01e092bee1a67cde52486cb3288fe0be2e86575fa0487c5e31a14f40e3214744"
Jul 09 16:15:29.617016 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.616990    2558 generic.go:356] "Generic (PLEG): container finished" podID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerID="096910f9d35e28dd2a0f052280ff33b1145229d3970af09a3b5ae011b18bce27" exitCode=0
Jul 09 16:15:29.617163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.617057    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv" event={"ID":"c34ab2f4-8edc-4723-8578-3edfdb0e7169","Type":"ContainerDied","Data":"096910f9d35e28dd2a0f052280ff33b1145229d3970af09a3b5ae011b18bce27"}
Jul 09 16:15:29.619320 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.619299    2558 generic.go:356] "Generic (PLEG): container finished" podID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerID="2e9acac8729c03aa5f2250e52689158c1451e1d012ddb7f89d5fa54b52e39f6f" exitCode=0
Jul 09 16:15:29.619421 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.619377    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4" event={"ID":"418616e9-35f0-4127-b3f3-9c3d582ff28c","Type":"ContainerDied","Data":"2e9acac8729c03aa5f2250e52689158c1451e1d012ddb7f89d5fa54b52e39f6f"}
Jul 09 16:15:29.744254 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.744231    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:29.843473 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.843448    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flwwq\" (UniqueName: \"kubernetes.io/projected/21ff865d-6fc8-4ec9-a663-5a6728b6c993-kube-api-access-flwwq\") pod \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") "
Jul 09 16:15:29.843619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.843482    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-bundle\") pod \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") "
Jul 09 16:15:29.843619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.843534    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-util\") pod \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\" (UID: \"21ff865d-6fc8-4ec9-a663-5a6728b6c993\") "
Jul 09 16:15:29.844085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.844058    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-bundle" (OuterVolumeSpecName: "bundle") pod "21ff865d-6fc8-4ec9-a663-5a6728b6c993" (UID: "21ff865d-6fc8-4ec9-a663-5a6728b6c993"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:29.845696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.845667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ff865d-6fc8-4ec9-a663-5a6728b6c993-kube-api-access-flwwq" (OuterVolumeSpecName: "kube-api-access-flwwq") pod "21ff865d-6fc8-4ec9-a663-5a6728b6c993" (UID: "21ff865d-6fc8-4ec9-a663-5a6728b6c993"). InnerVolumeSpecName "kube-api-access-flwwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:15:29.850530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.850483    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-util" (OuterVolumeSpecName: "util") pod "21ff865d-6fc8-4ec9-a663-5a6728b6c993" (UID: "21ff865d-6fc8-4ec9-a663-5a6728b6c993"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:29.944508 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.944445    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:29.944508 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.944478    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flwwq\" (UniqueName: \"kubernetes.io/projected/21ff865d-6fc8-4ec9-a663-5a6728b6c993-kube-api-access-flwwq\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:29.944508 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:29.944488    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21ff865d-6fc8-4ec9-a663-5a6728b6c993-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:30.625845 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.625816    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9"
Jul 09 16:15:30.625845 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.625829    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9" event={"ID":"21ff865d-6fc8-4ec9-a663-5a6728b6c993","Type":"ContainerDied","Data":"096be4748468113e85ecabfce71a2cf71b6910e162492f3619da03ac0de6c5be"}
Jul 09 16:15:30.626265 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.625901    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="096be4748468113e85ecabfce71a2cf71b6910e162492f3619da03ac0de6c5be"
Jul 09 16:15:30.758914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.758894    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:30.788265 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.788244    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:30.851397 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851376    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgngv\" (UniqueName: \"kubernetes.io/projected/c34ab2f4-8edc-4723-8578-3edfdb0e7169-kube-api-access-jgngv\") pod \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") "
Jul 09 16:15:30.851522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851409    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-util\") pod \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") "
Jul 09 16:15:30.851522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851441    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-util\") pod \"418616e9-35f0-4127-b3f3-9c3d582ff28c\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") "
Jul 09 16:15:30.851522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851488    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjgpv\" (UniqueName: \"kubernetes.io/projected/418616e9-35f0-4127-b3f3-9c3d582ff28c-kube-api-access-rjgpv\") pod \"418616e9-35f0-4127-b3f3-9c3d582ff28c\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") "
Jul 09 16:15:30.851522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851507    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-bundle\") pod \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\" (UID: \"c34ab2f4-8edc-4723-8578-3edfdb0e7169\") "
Jul 09 16:15:30.851733 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851532    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-bundle\") pod \"418616e9-35f0-4127-b3f3-9c3d582ff28c\" (UID: \"418616e9-35f0-4127-b3f3-9c3d582ff28c\") "
Jul 09 16:15:30.852011 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.851980    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-bundle" (OuterVolumeSpecName: "bundle") pod "c34ab2f4-8edc-4723-8578-3edfdb0e7169" (UID: "c34ab2f4-8edc-4723-8578-3edfdb0e7169"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:30.852128 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.852073    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-bundle" (OuterVolumeSpecName: "bundle") pod "418616e9-35f0-4127-b3f3-9c3d582ff28c" (UID: "418616e9-35f0-4127-b3f3-9c3d582ff28c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:30.853614 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.853588    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/418616e9-35f0-4127-b3f3-9c3d582ff28c-kube-api-access-rjgpv" (OuterVolumeSpecName: "kube-api-access-rjgpv") pod "418616e9-35f0-4127-b3f3-9c3d582ff28c" (UID: "418616e9-35f0-4127-b3f3-9c3d582ff28c"). InnerVolumeSpecName "kube-api-access-rjgpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:15:30.853929 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.853860    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c34ab2f4-8edc-4723-8578-3edfdb0e7169-kube-api-access-jgngv" (OuterVolumeSpecName: "kube-api-access-jgngv") pod "c34ab2f4-8edc-4723-8578-3edfdb0e7169" (UID: "c34ab2f4-8edc-4723-8578-3edfdb0e7169"). InnerVolumeSpecName "kube-api-access-jgngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:15:30.857242 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.857212    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-util" (OuterVolumeSpecName: "util") pod "c34ab2f4-8edc-4723-8578-3edfdb0e7169" (UID: "c34ab2f4-8edc-4723-8578-3edfdb0e7169"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:30.858046 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.858024    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-util" (OuterVolumeSpecName: "util") pod "418616e9-35f0-4127-b3f3-9c3d582ff28c" (UID: "418616e9-35f0-4127-b3f3-9c3d582ff28c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:15:30.953084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.953034    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:30.953084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.953054    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjgpv\" (UniqueName: \"kubernetes.io/projected/418616e9-35f0-4127-b3f3-9c3d582ff28c-kube-api-access-rjgpv\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:30.953084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.953064    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:30.953084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.953072    2558 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/418616e9-35f0-4127-b3f3-9c3d582ff28c-bundle\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:30.953084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.953080    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgngv\" (UniqueName: \"kubernetes.io/projected/c34ab2f4-8edc-4723-8578-3edfdb0e7169-kube-api-access-jgngv\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:30.953084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:30.953087    2558 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c34ab2f4-8edc-4723-8578-3edfdb0e7169-util\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:15:31.633173 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:31.633137    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv" event={"ID":"c34ab2f4-8edc-4723-8578-3edfdb0e7169","Type":"ContainerDied","Data":"c6bb59c7e19e2274eb65c0c27f3fde07c514509efb978e8aae63080e0f05fbcf"}
Jul 09 16:15:31.633591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:31.633217    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6bb59c7e19e2274eb65c0c27f3fde07c514509efb978e8aae63080e0f05fbcf"
Jul 09 16:15:31.633591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:31.633149    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv"
Jul 09 16:15:31.635431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:31.635411    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4"
Jul 09 16:15:31.635553 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:31.635408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4" event={"ID":"418616e9-35f0-4127-b3f3-9c3d582ff28c","Type":"ContainerDied","Data":"2e6519f66f5632ea30ba290024ed8d0da0638a32752e1a1bda86dcf96d207de1"}
Jul 09 16:15:31.635553 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:31.635540    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e6519f66f5632ea30ba290024ed8d0da0638a32752e1a1bda86dcf96d207de1"
Jul 09 16:15:40.493096 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493062    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"]
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493334    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493344    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493354    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493360    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493368    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493374    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493383    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493389    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493396    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493402    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493410    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493415    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493422    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493427    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493435    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493439    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493445    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493449    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493456    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493461    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="util"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493466    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493471    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493479    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493484    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="pull"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493530    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493537    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="c34ab2f4-8edc-4723-8578-3edfdb0e7169" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493543    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="418616e9-35f0-4127-b3f3-9c3d582ff28c" containerName="extract"
Jul 09 16:15:40.493567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.493549    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="21ff865d-6fc8-4ec9-a663-5a6728b6c993" containerName="extract"
Jul 09 16:15:40.495823 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.495806    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:40.499484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.499465    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-5462q\""
Jul 09 16:15:40.499871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.499852    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 09 16:15:40.512054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.512020    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"]
Jul 09 16:15:40.628579 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.628539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bpch\" (UniqueName: \"kubernetes.io/projected/cb03679a-9dd3-4449-b31f-79c34d2dcfdd-kube-api-access-2bpch\") pod \"dns-operator-controller-manager-864dd78c4c-cc6pf\" (UID: \"cb03679a-9dd3-4449-b31f-79c34d2dcfdd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:40.729418 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.729383    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bpch\" (UniqueName: \"kubernetes.io/projected/cb03679a-9dd3-4449-b31f-79c34d2dcfdd-kube-api-access-2bpch\") pod \"dns-operator-controller-manager-864dd78c4c-cc6pf\" (UID: \"cb03679a-9dd3-4449-b31f-79c34d2dcfdd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:40.749597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.749529    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bpch\" (UniqueName: \"kubernetes.io/projected/cb03679a-9dd3-4449-b31f-79c34d2dcfdd-kube-api-access-2bpch\") pod \"dns-operator-controller-manager-864dd78c4c-cc6pf\" (UID: \"cb03679a-9dd3-4449-b31f-79c34d2dcfdd\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:40.805553 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.805514    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:40.941679 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:40.941654    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"]
Jul 09 16:15:40.943869 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:40.943845    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb03679a_9dd3_4449_b31f_79c34d2dcfdd.slice/crio-c42361639e28e4b027dd8bc192cef2314758bbafbf59917dd34d7b49af7003df WatchSource:0}: Error finding container c42361639e28e4b027dd8bc192cef2314758bbafbf59917dd34d7b49af7003df: Status 404 returned error can't find the container with id c42361639e28e4b027dd8bc192cef2314758bbafbf59917dd34d7b49af7003df
Jul 09 16:15:41.684912 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:41.684863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf" event={"ID":"cb03679a-9dd3-4449-b31f-79c34d2dcfdd","Type":"ContainerStarted","Data":"c42361639e28e4b027dd8bc192cef2314758bbafbf59917dd34d7b49af7003df"}
Jul 09 16:15:43.705951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:43.705911    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf" event={"ID":"cb03679a-9dd3-4449-b31f-79c34d2dcfdd","Type":"ContainerStarted","Data":"14b9665530f781340312bde28a6d152763e5795406fc4bd1df978392af5b94f5"}
Jul 09 16:15:43.706370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:43.706007    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:43.723669 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:43.723619    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf" podStartSLOduration=1.59812662 podStartE2EDuration="3.723604389s" totalImagesPullingTime="2.125477769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:43.722191162 +0000 UTC m=+689.104007952" watchObservedRunningTime="2026-07-09 16:15:43.723604389 +0000 UTC m=+689.105421249"
Jul 09 16:15:44.546795 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.546756    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"]
Jul 09 16:15:44.552444 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.552415    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:44.555439 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.555407    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-zcmlw\""
Jul 09 16:15:44.564816 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.564789    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"]
Jul 09 16:15:44.660715 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.660674    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bclh\" (UniqueName: \"kubernetes.io/projected/413c821b-531a-475f-8dd4-9c7336799a16-kube-api-access-7bclh\") pod \"authorino-operator-75cc8d5cb7-czqxr\" (UID: \"413c821b-531a-475f-8dd4-9c7336799a16\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:44.761909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.761838    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bclh\" (UniqueName: \"kubernetes.io/projected/413c821b-531a-475f-8dd4-9c7336799a16-kube-api-access-7bclh\") pod \"authorino-operator-75cc8d5cb7-czqxr\" (UID: \"413c821b-531a-475f-8dd4-9c7336799a16\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:44.772816 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.772786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bclh\" (UniqueName: \"kubernetes.io/projected/413c821b-531a-475f-8dd4-9c7336799a16-kube-api-access-7bclh\") pod \"authorino-operator-75cc8d5cb7-czqxr\" (UID: \"413c821b-531a-475f-8dd4-9c7336799a16\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:44.864741 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:44.864706    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:45.006227 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:45.006188    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"]
Jul 09 16:15:45.008313 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:45.008288    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413c821b_531a_475f_8dd4_9c7336799a16.slice/crio-90bf869cb39c368cbc4a1f6901e23e1ca8c99e0ffde33f9afc3a598cb8a59166 WatchSource:0}: Error finding container 90bf869cb39c368cbc4a1f6901e23e1ca8c99e0ffde33f9afc3a598cb8a59166: Status 404 returned error can't find the container with id 90bf869cb39c368cbc4a1f6901e23e1ca8c99e0ffde33f9afc3a598cb8a59166
Jul 09 16:15:45.718366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:45.718321    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr" event={"ID":"413c821b-531a-475f-8dd4-9c7336799a16","Type":"ContainerStarted","Data":"90bf869cb39c368cbc4a1f6901e23e1ca8c99e0ffde33f9afc3a598cb8a59166"}
Jul 09 16:15:47.729257 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.729222    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr" event={"ID":"413c821b-531a-475f-8dd4-9c7336799a16","Type":"ContainerStarted","Data":"4d2bed84e54af22cb45fe282c175890bce64f9a5f212a20d886bda92817bdc45"}
Jul 09 16:15:47.729643 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.729279    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:47.757987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.757932    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr" podStartSLOduration=1.5216031220000001 podStartE2EDuration="3.757915261s" totalImagesPullingTime="2.236312139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:47.757400692 +0000 UTC m=+693.139217498" watchObservedRunningTime="2026-07-09 16:15:47.757915261 +0000 UTC m=+693.139732049"
Jul 09 16:15:47.765269 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.765239    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"]
Jul 09 16:15:47.768614 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.768593    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:15:47.771575 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.771556    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-npmxx\""
Jul 09 16:15:47.785169 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.783384    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"]
Jul 09 16:15:47.890384 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.890341    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzxf\" (UniqueName: \"kubernetes.io/projected/459e1b9a-0ba2-4454-bd24-02243adcdd2f-kube-api-access-zqzxf\") pod \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" (UID: \"459e1b9a-0ba2-4454-bd24-02243adcdd2f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:15:47.991452 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:47.991355    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzxf\" (UniqueName: \"kubernetes.io/projected/459e1b9a-0ba2-4454-bd24-02243adcdd2f-kube-api-access-zqzxf\") pod \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" (UID: \"459e1b9a-0ba2-4454-bd24-02243adcdd2f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:15:48.010827 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:48.010790    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzxf\" (UniqueName: \"kubernetes.io/projected/459e1b9a-0ba2-4454-bd24-02243adcdd2f-kube-api-access-zqzxf\") pod \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" (UID: \"459e1b9a-0ba2-4454-bd24-02243adcdd2f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:15:48.078478 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:48.078443    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:15:48.224863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:48.224837    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"]
Jul 09 16:15:48.226509 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:48.226476    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod459e1b9a_0ba2_4454_bd24_02243adcdd2f.slice/crio-ed139b7b6be444ac94fbef05d8319c2030f6d7bea43af90ae9aeb8446903276f WatchSource:0}: Error finding container ed139b7b6be444ac94fbef05d8319c2030f6d7bea43af90ae9aeb8446903276f: Status 404 returned error can't find the container with id ed139b7b6be444ac94fbef05d8319c2030f6d7bea43af90ae9aeb8446903276f
Jul 09 16:15:48.736163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:48.736125    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd" event={"ID":"459e1b9a-0ba2-4454-bd24-02243adcdd2f","Type":"ContainerStarted","Data":"ed139b7b6be444ac94fbef05d8319c2030f6d7bea43af90ae9aeb8446903276f"}
Jul 09 16:15:50.748100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:50.748014    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd" event={"ID":"459e1b9a-0ba2-4454-bd24-02243adcdd2f","Type":"ContainerStarted","Data":"2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf"}
Jul 09 16:15:50.748569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:50.748144    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:15:50.765760 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:50.765714    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd" podStartSLOduration=1.667212329 podStartE2EDuration="3.765697888s" totalImagesPullingTime="2.098485559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:50.764319599 +0000 UTC m=+696.146136389" watchObservedRunningTime="2026-07-09 16:15:50.765697888 +0000 UTC m=+696.147514676"
Jul 09 16:15:52.363859 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.363823    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"]
Jul 09 16:15:52.367185 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.367165    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.370331 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.370310    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cvhzc\""
Jul 09 16:15:52.388871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.388836    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"]
Jul 09 16:15:52.534934 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.534889    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12852fc9-8959-47f3-8137-887d874de0d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.535115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.534944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdjt9\" (UniqueName: \"kubernetes.io/projected/12852fc9-8959-47f3-8137-887d874de0d5-kube-api-access-hdjt9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.635497 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.635408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12852fc9-8959-47f3-8137-887d874de0d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.635497 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.635448    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdjt9\" (UniqueName: \"kubernetes.io/projected/12852fc9-8959-47f3-8137-887d874de0d5-kube-api-access-hdjt9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.635824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.635802    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12852fc9-8959-47f3-8137-887d874de0d5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.647468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.647436    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdjt9\" (UniqueName: \"kubernetes.io/projected/12852fc9-8959-47f3-8137-887d874de0d5-kube-api-access-hdjt9\") pod \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.678122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.678085    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:52.820492 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:52.820461    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"]
Jul 09 16:15:52.822334 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:15:52.822306    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12852fc9_8959_47f3_8137_887d874de0d5.slice/crio-042b8e3b7a2183b33d3721dc4df9074bf1ed2ebfcd742fee58256b82bb2a4d85 WatchSource:0}: Error finding container 042b8e3b7a2183b33d3721dc4df9074bf1ed2ebfcd742fee58256b82bb2a4d85: Status 404 returned error can't find the container with id 042b8e3b7a2183b33d3721dc4df9074bf1ed2ebfcd742fee58256b82bb2a4d85
Jul 09 16:15:53.764766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:53.764721    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47" event={"ID":"12852fc9-8959-47f3-8137-887d874de0d5","Type":"ContainerStarted","Data":"042b8e3b7a2183b33d3721dc4df9074bf1ed2ebfcd742fee58256b82bb2a4d85"}
Jul 09 16:15:54.713804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:54.713773    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-cc6pf"
Jul 09 16:15:58.739412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:58.739383    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-czqxr"
Jul 09 16:15:59.797092 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:59.796997    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47" event={"ID":"12852fc9-8959-47f3-8137-887d874de0d5","Type":"ContainerStarted","Data":"30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128"}
Jul 09 16:15:59.797440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:59.797105    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:15:59.822828 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:15:59.822779    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47" podStartSLOduration=1.283932156 podStartE2EDuration="7.822763497s" totalImagesPullingTime="6.538831341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:15:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:15:59.822258619 +0000 UTC m=+705.204075406" watchObservedRunningTime="2026-07-09 16:15:59.822763497 +0000 UTC m=+705.204580284"
Jul 09 16:16:01.755820 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:01.755789    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:10.804295 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:10.804217    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.590656 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.590624    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"]
Jul 09 16:16:12.591059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.590808    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47" podUID="12852fc9-8959-47f3-8137-887d874de0d5" containerName="manager" containerID="cri-o://30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128" gracePeriod=2
Jul 09 16:16:12.602379 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.602349    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"]
Jul 09 16:16:12.607828 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.607640    2558 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"12852fc9-8959-47f3-8137-887d874de0d5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-bgp47\": pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.620460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.620430    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"]
Jul 09 16:16:12.620796 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.620782    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12852fc9-8959-47f3-8137-887d874de0d5" containerName="manager"
Jul 09 16:16:12.620840 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.620799    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="12852fc9-8959-47f3-8137-887d874de0d5" containerName="manager"
Jul 09 16:16:12.620889 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.620863    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="12852fc9-8959-47f3-8137-887d874de0d5" containerName="manager"
Jul 09 16:16:12.623494 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.623477    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"]
Jul 09 16:16:12.623644 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.623621    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.623717 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.623627    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" containerName="manager" containerID="cri-o://2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf" gracePeriod=2
Jul 09 16:16:12.627547 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.627100    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.627823 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.627803    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"]
Jul 09 16:16:12.630067 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.630043    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.632312 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.632290    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.635722 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.635693    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"]
Jul 09 16:16:12.635890 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.635851    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.646645 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.646621    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"]
Jul 09 16:16:12.646971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.646957    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" containerName="manager"
Jul 09 16:16:12.646971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.646972    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" containerName="manager"
Jul 09 16:16:12.647109 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.647044    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" containerName="manager"
Jul 09 16:16:12.649773 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.649757    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:12.659145 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.659122    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"]
Jul 09 16:16:12.683359 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.683328    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.685659 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.685637    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.691000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.690973    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b488b91-1bc2-4e8f-8298-021291088a3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcvch\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.691088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.691004    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/cc8e8027-3a55-4e3a-8454-fc23ecc7abbc-kube-api-access-2ltmh\") pod \"limitador-operator-controller-manager-859d7b78bd-hc5c4\" (UID: \"cc8e8027-3a55-4e3a-8454-fc23ecc7abbc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:12.691088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.691026    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hx4b\" (UniqueName: \"kubernetes.io/projected/9b488b91-1bc2-4e8f-8298-021291088a3f-kube-api-access-4hx4b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcvch\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.792986 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.792499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b488b91-1bc2-4e8f-8298-021291088a3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcvch\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.792986 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.792563    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/cc8e8027-3a55-4e3a-8454-fc23ecc7abbc-kube-api-access-2ltmh\") pod \"limitador-operator-controller-manager-859d7b78bd-hc5c4\" (UID: \"cc8e8027-3a55-4e3a-8454-fc23ecc7abbc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:12.792986 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.792600    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hx4b\" (UniqueName: \"kubernetes.io/projected/9b488b91-1bc2-4e8f-8298-021291088a3f-kube-api-access-4hx4b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcvch\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.793216 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.793015    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b488b91-1bc2-4e8f-8298-021291088a3f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcvch\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.805707 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.804858    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ltmh\" (UniqueName: \"kubernetes.io/projected/cc8e8027-3a55-4e3a-8454-fc23ecc7abbc-kube-api-access-2ltmh\") pod \"limitador-operator-controller-manager-859d7b78bd-hc5c4\" (UID: \"cc8e8027-3a55-4e3a-8454-fc23ecc7abbc\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:12.805707 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.805682    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hx4b\" (UniqueName: \"kubernetes.io/projected/9b488b91-1bc2-4e8f-8298-021291088a3f-kube-api-access-4hx4b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-wcvch\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:12.851494 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.851470    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.854101 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.854077    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.856256 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.856229    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.857385 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.857365    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.859722 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.859689    2558 generic.go:356] "Generic (PLEG): container finished" podID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" containerID="2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf" exitCode=0
Jul 09 16:16:12.859813 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.859735    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.859873 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.859805    2558 scope.go:117] "RemoveContainer" containerID="2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf"
Jul 09 16:16:12.859873 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.859840    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.861138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.861119    2558 generic.go:356] "Generic (PLEG): container finished" podID="12852fc9-8959-47f3-8137-887d874de0d5" containerID="30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128" exitCode=0
Jul 09 16:16:12.861224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.861160    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.862589 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.862565    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.866202 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.866179    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:12.868278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.868265    2558 scope.go:117] "RemoveContainer" containerID="2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf"
Jul 09 16:16:12.868421 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.868394    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:12.868585 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.868561    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf\": container with ID starting with 2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf not found: ID does not exist" containerID="2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf"
Jul 09 16:16:12.868633 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.868596    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf"} err="failed to get container status \"2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf\": rpc error: code = NotFound desc = could not find container \"2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf\": container with ID starting with 2d30fad6d55e28cbd59e20c650a43cacb3e8c95bc5c6528477285e8110992ecf not found: ID does not exist"
Jul 09 16:16:12.868633 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.868616    2558 scope.go:117] "RemoveContainer" containerID="30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128"
Jul 09 16:16:12.876176 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.876155    2558 scope.go:117] "RemoveContainer" containerID="30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128"
Jul 09 16:16:12.876448 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:12.876432    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128\": container with ID starting with 30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128 not found: ID does not exist" containerID="30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128"
Jul 09 16:16:12.876497 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.876455    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128"} err="failed to get container status \"30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128\": rpc error: code = NotFound desc = could not find container \"30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128\": container with ID starting with 30da9aedfaba8455570c25541b91d3d2f5d52345044b6c3b9508b57c98847128 not found: ID does not exist"
Jul 09 16:16:12.893779 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.893763    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdjt9\" (UniqueName: \"kubernetes.io/projected/12852fc9-8959-47f3-8137-887d874de0d5-kube-api-access-hdjt9\") pod \"12852fc9-8959-47f3-8137-887d874de0d5\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") "
Jul 09 16:16:12.893850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.893798    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12852fc9-8959-47f3-8137-887d874de0d5-extensions-socket-volume\") pod \"12852fc9-8959-47f3-8137-887d874de0d5\" (UID: \"12852fc9-8959-47f3-8137-887d874de0d5\") "
Jul 09 16:16:12.893929 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.893856    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqzxf\" (UniqueName: \"kubernetes.io/projected/459e1b9a-0ba2-4454-bd24-02243adcdd2f-kube-api-access-zqzxf\") pod \"459e1b9a-0ba2-4454-bd24-02243adcdd2f\" (UID: \"459e1b9a-0ba2-4454-bd24-02243adcdd2f\") "
Jul 09 16:16:12.894360 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.894337    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/12852fc9-8959-47f3-8137-887d874de0d5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "12852fc9-8959-47f3-8137-887d874de0d5" (UID: "12852fc9-8959-47f3-8137-887d874de0d5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:16:12.895753 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.895732    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12852fc9-8959-47f3-8137-887d874de0d5-kube-api-access-hdjt9" (OuterVolumeSpecName: "kube-api-access-hdjt9") pod "12852fc9-8959-47f3-8137-887d874de0d5" (UID: "12852fc9-8959-47f3-8137-887d874de0d5"). InnerVolumeSpecName "kube-api-access-hdjt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:16:12.895813 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.895795    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459e1b9a-0ba2-4454-bd24-02243adcdd2f-kube-api-access-zqzxf" (OuterVolumeSpecName: "kube-api-access-zqzxf") pod "459e1b9a-0ba2-4454-bd24-02243adcdd2f" (UID: "459e1b9a-0ba2-4454-bd24-02243adcdd2f"). InnerVolumeSpecName "kube-api-access-zqzxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:16:12.994574 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.994552    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzxf\" (UniqueName: \"kubernetes.io/projected/459e1b9a-0ba2-4454-bd24-02243adcdd2f-kube-api-access-zqzxf\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:16:12.994574 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.994570    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdjt9\" (UniqueName: \"kubernetes.io/projected/12852fc9-8959-47f3-8137-887d874de0d5-kube-api-access-hdjt9\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:16:12.994693 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:12.994581    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/12852fc9-8959-47f3-8137-887d874de0d5-extensions-socket-volume\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:16:13.036552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.036519    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:13.041641 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.041627    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:13.176585 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:13.176552    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-bgp47\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="12852fc9-8959-47f3-8137-887d874de0d5" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-bgp47"
Jul 09 16:16:13.178933 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:13.178826    2558 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-5rpqd\" is forbidden: User \"system:node:ip-10-0-133-91.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-133-91.ec2.internal' and this object" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-5rpqd"
Jul 09 16:16:13.179205 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.179178    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12852fc9-8959-47f3-8137-887d874de0d5" path="/var/lib/kubelet/pods/12852fc9-8959-47f3-8137-887d874de0d5/volumes"
Jul 09 16:16:13.179664 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.179644    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="459e1b9a-0ba2-4454-bd24-02243adcdd2f" path="/var/lib/kubelet/pods/459e1b9a-0ba2-4454-bd24-02243adcdd2f/volumes"
Jul 09 16:16:13.182246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.182158    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"]
Jul 09 16:16:13.184759 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:16:13.184737    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b488b91_1bc2_4e8f_8298_021291088a3f.slice/crio-ccd6bdd9b9c33b1e78e0d8044511819f382c296c4422688cff2481a9651b1cc1 WatchSource:0}: Error finding container ccd6bdd9b9c33b1e78e0d8044511819f382c296c4422688cff2481a9651b1cc1: Status 404 returned error can't find the container with id ccd6bdd9b9c33b1e78e0d8044511819f382c296c4422688cff2481a9651b1cc1
Jul 09 16:16:13.203211 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.203194    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"]
Jul 09 16:16:13.205084 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:16:13.205064    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8e8027_3a55_4e3a_8454_fc23ecc7abbc.slice/crio-59b7f6d46cd929c42cbfeec84e6d874db4dc96bf89071bd5f7b115b7c674fad8 WatchSource:0}: Error finding container 59b7f6d46cd929c42cbfeec84e6d874db4dc96bf89071bd5f7b115b7c674fad8: Status 404 returned error can't find the container with id 59b7f6d46cd929c42cbfeec84e6d874db4dc96bf89071bd5f7b115b7c674fad8
Jul 09 16:16:13.871554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.871469    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4" event={"ID":"cc8e8027-3a55-4e3a-8454-fc23ecc7abbc","Type":"ContainerStarted","Data":"d6491fea4f0c1cf72baeb4e88213962495a0634be4970497c69314d0f560cfea"}
Jul 09 16:16:13.871554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.871524    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4" event={"ID":"cc8e8027-3a55-4e3a-8454-fc23ecc7abbc","Type":"ContainerStarted","Data":"59b7f6d46cd929c42cbfeec84e6d874db4dc96bf89071bd5f7b115b7c674fad8"}
Jul 09 16:16:13.871554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.871549    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:13.873785 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.873760    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch" event={"ID":"9b488b91-1bc2-4e8f-8298-021291088a3f","Type":"ContainerStarted","Data":"f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848"}
Jul 09 16:16:13.873952 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.873805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch" event={"ID":"9b488b91-1bc2-4e8f-8298-021291088a3f","Type":"ContainerStarted","Data":"ccd6bdd9b9c33b1e78e0d8044511819f382c296c4422688cff2481a9651b1cc1"}
Jul 09 16:16:13.873952 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.873841    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:13.898978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.898939    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4" podStartSLOduration=1.898925593 podStartE2EDuration="1.898925593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:13.896964146 +0000 UTC m=+719.278780934" watchObservedRunningTime="2026-07-09 16:16:13.898925593 +0000 UTC m=+719.280742359"
Jul 09 16:16:13.925366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:13.925324    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch" podStartSLOduration=1.925310098 podStartE2EDuration="1.925310098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:13.923589191 +0000 UTC m=+719.305405980" watchObservedRunningTime="2026-07-09 16:16:13.925310098 +0000 UTC m=+719.307126919"
Jul 09 16:16:24.885000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:24.884968    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:24.885467 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:24.885168    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-hc5c4"
Jul 09 16:16:43.596571 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.596520    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"]
Jul 09 16:16:43.597309 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.596747    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch" podUID="9b488b91-1bc2-4e8f-8298-021291088a3f" containerName="manager" containerID="cri-o://f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848" gracePeriod=10
Jul 09 16:16:43.835668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.835642    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"]
Jul 09 16:16:43.837870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.837856    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:43.842686 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.842662    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:43.855845 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.855780    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"]
Jul 09 16:16:43.969041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.969005    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hx4b\" (UniqueName: \"kubernetes.io/projected/9b488b91-1bc2-4e8f-8298-021291088a3f-kube-api-access-4hx4b\") pod \"9b488b91-1bc2-4e8f-8298-021291088a3f\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") "
Jul 09 16:16:43.969234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.969098    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b488b91-1bc2-4e8f-8298-021291088a3f-extensions-socket-volume\") pod \"9b488b91-1bc2-4e8f-8298-021291088a3f\" (UID: \"9b488b91-1bc2-4e8f-8298-021291088a3f\") "
Jul 09 16:16:43.969305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.969280    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcwct\" (UniqueName: \"kubernetes.io/projected/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-kube-api-access-wcwct\") pod \"kuadrant-operator-controller-manager-86878c769f-jfjc4\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:43.969361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.969330    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jfjc4\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:43.969517 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.969490    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b488b91-1bc2-4e8f-8298-021291088a3f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9b488b91-1bc2-4e8f-8298-021291088a3f" (UID: "9b488b91-1bc2-4e8f-8298-021291088a3f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:16:43.971267 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:43.971244    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b488b91-1bc2-4e8f-8298-021291088a3f-kube-api-access-4hx4b" (OuterVolumeSpecName: "kube-api-access-4hx4b") pod "9b488b91-1bc2-4e8f-8298-021291088a3f" (UID: "9b488b91-1bc2-4e8f-8298-021291088a3f"). InnerVolumeSpecName "kube-api-access-4hx4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:16:44.030933 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.030898    2558 generic.go:356] "Generic (PLEG): container finished" podID="9b488b91-1bc2-4e8f-8298-021291088a3f" containerID="f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848" exitCode=0
Jul 09 16:16:44.031099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.030941    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"
Jul 09 16:16:44.031099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.030975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch" event={"ID":"9b488b91-1bc2-4e8f-8298-021291088a3f","Type":"ContainerDied","Data":"f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848"}
Jul 09 16:16:44.031099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.031033    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch" event={"ID":"9b488b91-1bc2-4e8f-8298-021291088a3f","Type":"ContainerDied","Data":"ccd6bdd9b9c33b1e78e0d8044511819f382c296c4422688cff2481a9651b1cc1"}
Jul 09 16:16:44.031099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.031060    2558 scope.go:117] "RemoveContainer" containerID="f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848"
Jul 09 16:16:44.042576 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.041687    2558 scope.go:117] "RemoveContainer" containerID="f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848"
Jul 09 16:16:44.042576 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:16:44.042115    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848\": container with ID starting with f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848 not found: ID does not exist" containerID="f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848"
Jul 09 16:16:44.042576 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.042146    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848"} err="failed to get container status \"f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848\": rpc error: code = NotFound desc = could not find container \"f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848\": container with ID starting with f60f4c8aa5372ea3417583c13b2bafc98debeab503b9ef1f0400d281eb785848 not found: ID does not exist"
Jul 09 16:16:44.055433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.055412    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"]
Jul 09 16:16:44.062248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.062226    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-wcvch"]
Jul 09 16:16:44.070559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.070532    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcwct\" (UniqueName: \"kubernetes.io/projected/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-kube-api-access-wcwct\") pod \"kuadrant-operator-controller-manager-86878c769f-jfjc4\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:44.070658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.070568    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jfjc4\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:44.070658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.070625    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9b488b91-1bc2-4e8f-8298-021291088a3f-extensions-socket-volume\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:16:44.070658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.070635    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hx4b\" (UniqueName: \"kubernetes.io/projected/9b488b91-1bc2-4e8f-8298-021291088a3f-kube-api-access-4hx4b\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:16:44.070972 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.070953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jfjc4\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:44.079628 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.079611    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcwct\" (UniqueName: \"kubernetes.io/projected/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-kube-api-access-wcwct\") pod \"kuadrant-operator-controller-manager-86878c769f-jfjc4\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:44.149051 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.148970    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:44.285593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:44.285570    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"]
Jul 09 16:16:44.287579 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:16:44.287554    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3db92a6f_2e9d_4dae_8a4c_8535dddf152d.slice/crio-bb75db850811e01de6b588e581a78d774fffa07600391c8760d4c20d6487c391 WatchSource:0}: Error finding container bb75db850811e01de6b588e581a78d774fffa07600391c8760d4c20d6487c391: Status 404 returned error can't find the container with id bb75db850811e01de6b588e581a78d774fffa07600391c8760d4c20d6487c391
Jul 09 16:16:45.043386 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:45.043345    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4" event={"ID":"3db92a6f-2e9d-4dae-8a4c-8535dddf152d","Type":"ContainerStarted","Data":"7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05"}
Jul 09 16:16:45.043802 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:45.043423    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4" event={"ID":"3db92a6f-2e9d-4dae-8a4c-8535dddf152d","Type":"ContainerStarted","Data":"bb75db850811e01de6b588e581a78d774fffa07600391c8760d4c20d6487c391"}
Jul 09 16:16:45.043802 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:45.043500    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:45.065858 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:45.065805    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4" podStartSLOduration=2.06578935 podStartE2EDuration="2.06578935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:16:45.065426138 +0000 UTC m=+750.447242926" watchObservedRunningTime="2026-07-09 16:16:45.06578935 +0000 UTC m=+750.447606139"
Jul 09 16:16:45.177782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:45.177741    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b488b91-1bc2-4e8f-8298-021291088a3f" path="/var/lib/kubelet/pods/9b488b91-1bc2-4e8f-8298-021291088a3f/volumes"
Jul 09 16:16:56.049577 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:56.049542    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:16:59.915511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.915475    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"]
Jul 09 16:16:59.915872 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.915808    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b488b91-1bc2-4e8f-8298-021291088a3f" containerName="manager"
Jul 09 16:16:59.915872 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.915818    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b488b91-1bc2-4e8f-8298-021291088a3f" containerName="manager"
Jul 09 16:16:59.915975 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.915893    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b488b91-1bc2-4e8f-8298-021291088a3f" containerName="manager"
Jul 09 16:16:59.918445 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.918427    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.921609 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.921582    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-c46tv\""
Jul 09 16:16:59.929832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.929800    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"]
Jul 09 16:16:59.998297 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998255    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998297 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998509 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998342    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7851200e-a8e4-4934-8d53-0d483d116ec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998509 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998412    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998509 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998457    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998509 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998485    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998525    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998547    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:16:59.998634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:16:59.998587    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6k9m\" (UniqueName: \"kubernetes.io/projected/7851200e-a8e4-4934-8d53-0d483d116ec2-kube-api-access-d6k9m\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.099924 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.099862    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6k9m\" (UniqueName: \"kubernetes.io/projected/7851200e-a8e4-4934-8d53-0d483d116ec2-kube-api-access-d6k9m\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.099924 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.099927    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.099954    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.099976    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7851200e-a8e4-4934-8d53-0d483d116ec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100005    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100213    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100276    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100305    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100472    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100557 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100502    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100707 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100684    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100709    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.100819 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.100766    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7851200e-a8e4-4934-8d53-0d483d116ec2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.102639 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.102617    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.102773 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.102758    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.109974 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.109943    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7851200e-a8e4-4934-8d53-0d483d116ec2-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.110178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.110157    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6k9m\" (UniqueName: \"kubernetes.io/projected/7851200e-a8e4-4934-8d53-0d483d116ec2-kube-api-access-d6k9m\") pod \"maas-default-gateway-openshift-default-5896d4dd49-nls8r\" (UID: \"7851200e-a8e4-4934-8d53-0d483d116ec2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.230719 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.230604    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:00.381581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.381549    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"]
Jul 09 16:17:00.383570 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:17:00.383538    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7851200e_a8e4_4934_8d53_0d483d116ec2.slice/crio-f8bfd9cb73b643564a4578509c9a4ae2afb160b99477989eddb6a2d59dfba0d6 WatchSource:0}: Error finding container f8bfd9cb73b643564a4578509c9a4ae2afb160b99477989eddb6a2d59dfba0d6: Status 404 returned error can't find the container with id f8bfd9cb73b643564a4578509c9a4ae2afb160b99477989eddb6a2d59dfba0d6
Jul 09 16:17:00.385825 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.385789    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:17:00.385962 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.385868    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:17:00.385962 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:00.385931    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:17:01.124174 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:01.124130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" event={"ID":"7851200e-a8e4-4934-8d53-0d483d116ec2","Type":"ContainerStarted","Data":"bf9841c45c0a24ff4c9326a99c84a3fe153c1b3b21a7428eae51160e20fbd82b"}
Jul 09 16:17:01.124597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:01.124193    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" event={"ID":"7851200e-a8e4-4934-8d53-0d483d116ec2","Type":"ContainerStarted","Data":"f8bfd9cb73b643564a4578509c9a4ae2afb160b99477989eddb6a2d59dfba0d6"}
Jul 09 16:17:01.149237 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:01.149174    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" podStartSLOduration=2.149153214 podStartE2EDuration="2.149153214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:01.147101063 +0000 UTC m=+766.528917851" watchObservedRunningTime="2026-07-09 16:17:01.149153214 +0000 UTC m=+766.530970002"
Jul 09 16:17:01.231372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:01.231331    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:01.236207 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:01.236180    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:02.130401 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:02.130368    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:02.131455 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:02.131430    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:17:05.336703 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.336665    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-s8mxc"]
Jul 09 16:17:05.339136 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.339113    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:05.342138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.342102    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k4nvx\""
Jul 09 16:17:05.349113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.349080    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8mxc"]
Jul 09 16:17:05.445396 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.445348    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmgf\" (UniqueName: \"kubernetes.io/projected/7071d512-5818-44cb-9153-3cafb073d6ee-kube-api-access-2xmgf\") pod \"authorino-58cd955675-s8mxc\" (UID: \"7071d512-5818-44cb-9153-3cafb073d6ee\") " pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:05.535640 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.535596    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-skgln"]
Jul 09 16:17:05.538299 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.538264    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:05.545290 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.545240    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-skgln"]
Jul 09 16:17:05.545989 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.545953    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmgf\" (UniqueName: \"kubernetes.io/projected/7071d512-5818-44cb-9153-3cafb073d6ee-kube-api-access-2xmgf\") pod \"authorino-58cd955675-s8mxc\" (UID: \"7071d512-5818-44cb-9153-3cafb073d6ee\") " pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:05.561696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.561662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmgf\" (UniqueName: \"kubernetes.io/projected/7071d512-5818-44cb-9153-3cafb073d6ee-kube-api-access-2xmgf\") pod \"authorino-58cd955675-s8mxc\" (UID: \"7071d512-5818-44cb-9153-3cafb073d6ee\") " pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:05.647210 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.647110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4jwf\" (UniqueName: \"kubernetes.io/projected/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f-kube-api-access-c4jwf\") pod \"authorino-75d556bd77-skgln\" (UID: \"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f\") " pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:05.650976 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.650944    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:05.748489 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.748449    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4jwf\" (UniqueName: \"kubernetes.io/projected/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f-kube-api-access-c4jwf\") pod \"authorino-75d556bd77-skgln\" (UID: \"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f\") " pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:05.757898 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.757845    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4jwf\" (UniqueName: \"kubernetes.io/projected/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f-kube-api-access-c4jwf\") pod \"authorino-75d556bd77-skgln\" (UID: \"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f\") " pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:05.801316 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.801286    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8mxc"]
Jul 09 16:17:05.803943 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:17:05.803907    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7071d512_5818_44cb_9153_3cafb073d6ee.slice/crio-91f9c02b07bcc30ec892b87a0fd1c3b63bd587d611c69ef02860c90ad72a6105 WatchSource:0}: Error finding container 91f9c02b07bcc30ec892b87a0fd1c3b63bd587d611c69ef02860c90ad72a6105: Status 404 returned error can't find the container with id 91f9c02b07bcc30ec892b87a0fd1c3b63bd587d611c69ef02860c90ad72a6105
Jul 09 16:17:05.849584 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.849547    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:05.992666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:05.992623    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-skgln"]
Jul 09 16:17:05.996707 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:17:05.996676    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda27af3c4_0bc9_4c7c_9135_7bf977b0ee9f.slice/crio-18d62c03696a441e7608b2222693d86a52d6d760e4d7d6fbda5bdca9c9ce23e8 WatchSource:0}: Error finding container 18d62c03696a441e7608b2222693d86a52d6d760e4d7d6fbda5bdca9c9ce23e8: Status 404 returned error can't find the container with id 18d62c03696a441e7608b2222693d86a52d6d760e4d7d6fbda5bdca9c9ce23e8
Jul 09 16:17:06.153454 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:06.153342    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-skgln" event={"ID":"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f","Type":"ContainerStarted","Data":"18d62c03696a441e7608b2222693d86a52d6d760e4d7d6fbda5bdca9c9ce23e8"}
Jul 09 16:17:06.154616 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:06.154585    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8mxc" event={"ID":"7071d512-5818-44cb-9153-3cafb073d6ee","Type":"ContainerStarted","Data":"91f9c02b07bcc30ec892b87a0fd1c3b63bd587d611c69ef02860c90ad72a6105"}
Jul 09 16:17:10.179920 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:10.179863    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8mxc" event={"ID":"7071d512-5818-44cb-9153-3cafb073d6ee","Type":"ContainerStarted","Data":"3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e"}
Jul 09 16:17:10.181441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:10.181408    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-skgln" event={"ID":"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f","Type":"ContainerStarted","Data":"87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b"}
Jul 09 16:17:10.217209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:10.217155    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-s8mxc" podStartSLOduration=1.420979851 podStartE2EDuration="5.217137949s" totalImagesPullingTime="3.796158098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:10.215850185 +0000 UTC m=+775.597666972" watchObservedRunningTime="2026-07-09 16:17:10.217137949 +0000 UTC m=+775.598954736"
Jul 09 16:17:10.263871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:10.263811    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-skgln" podStartSLOduration=1.649225402 podStartE2EDuration="5.263792114s" totalImagesPullingTime="3.614566712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:17:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:10.261825435 +0000 UTC m=+775.643642223" watchObservedRunningTime="2026-07-09 16:17:10.263792114 +0000 UTC m=+775.645608902"
Jul 09 16:17:10.331820 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:10.331783    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8mxc"]
Jul 09 16:17:12.191329 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:12.191282    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-s8mxc" podUID="7071d512-5818-44cb-9153-3cafb073d6ee" containerName="authorino" containerID="cri-o://3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e" gracePeriod=30
Jul 09 16:17:12.460152 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:12.460121    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:12.509853 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:12.509812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xmgf\" (UniqueName: \"kubernetes.io/projected/7071d512-5818-44cb-9153-3cafb073d6ee-kube-api-access-2xmgf\") pod \"7071d512-5818-44cb-9153-3cafb073d6ee\" (UID: \"7071d512-5818-44cb-9153-3cafb073d6ee\") "
Jul 09 16:17:12.512128 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:12.512097    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7071d512-5818-44cb-9153-3cafb073d6ee-kube-api-access-2xmgf" (OuterVolumeSpecName: "kube-api-access-2xmgf") pod "7071d512-5818-44cb-9153-3cafb073d6ee" (UID: "7071d512-5818-44cb-9153-3cafb073d6ee"). InnerVolumeSpecName "kube-api-access-2xmgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:17:12.611100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:12.611050    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmgf\" (UniqueName: \"kubernetes.io/projected/7071d512-5818-44cb-9153-3cafb073d6ee-kube-api-access-2xmgf\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:17:13.197224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.197182    2558 generic.go:356] "Generic (PLEG): container finished" podID="7071d512-5818-44cb-9153-3cafb073d6ee" containerID="3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e" exitCode=0
Jul 09 16:17:13.197631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.197238    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8mxc" event={"ID":"7071d512-5818-44cb-9153-3cafb073d6ee","Type":"ContainerDied","Data":"3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e"}
Jul 09 16:17:13.197631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.197245    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-s8mxc"
Jul 09 16:17:13.197631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.197281    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-s8mxc" event={"ID":"7071d512-5818-44cb-9153-3cafb073d6ee","Type":"ContainerDied","Data":"91f9c02b07bcc30ec892b87a0fd1c3b63bd587d611c69ef02860c90ad72a6105"}
Jul 09 16:17:13.197631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.197304    2558 scope.go:117] "RemoveContainer" containerID="3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e"
Jul 09 16:17:13.207261 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.207236    2558 scope.go:117] "RemoveContainer" containerID="3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e"
Jul 09 16:17:13.207611 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:17:13.207589    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e\": container with ID starting with 3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e not found: ID does not exist" containerID="3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e"
Jul 09 16:17:13.207667 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.207624    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e"} err="failed to get container status \"3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e\": rpc error: code = NotFound desc = could not find container \"3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e\": container with ID starting with 3f782f890392d20324b63ba47c3bb59c07d8f6cd752fb34062bc236db3b3c30e not found: ID does not exist"
Jul 09 16:17:13.218541 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.218497    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8mxc"]
Jul 09 16:17:13.224525 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:13.224487    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-s8mxc"]
Jul 09 16:17:15.178752 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:15.178717    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7071d512-5818-44cb-9153-3cafb073d6ee" path="/var/lib/kubelet/pods/7071d512-5818-44cb-9153-3cafb073d6ee/volumes"
Jul 09 16:17:35.946441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:35.946353    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-45f4r"]
Jul 09 16:17:35.946950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:35.946686    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7071d512-5818-44cb-9153-3cafb073d6ee" containerName="authorino"
Jul 09 16:17:35.946950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:35.946697    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7071d512-5818-44cb-9153-3cafb073d6ee" containerName="authorino"
Jul 09 16:17:35.946950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:35.946746    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7071d512-5818-44cb-9153-3cafb073d6ee" containerName="authorino"
Jul 09 16:17:35.950296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:35.950275    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:35.956588 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:35.956558    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-45f4r"]
Jul 09 16:17:36.124470 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.124428    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9mpn\" (UniqueName: \"kubernetes.io/projected/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d-kube-api-access-z9mpn\") pod \"authorino-5d4b67c446-45f4r\" (UID: \"8e99f314-66d5-4684-aec9-fe6b5c1b2e9d\") " pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:36.205179 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.205095    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-45f4r"]
Jul 09 16:17:36.205391 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:17:36.205370    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-z9mpn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-45f4r" podUID="8e99f314-66d5-4684-aec9-fe6b5c1b2e9d"
Jul 09 16:17:36.221150 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.221106    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7d7bcdc9c5-lssp7"]
Jul 09 16:17:36.226043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.226008    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9mpn\" (UniqueName: \"kubernetes.io/projected/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d-kube-api-access-z9mpn\") pod \"authorino-5d4b67c446-45f4r\" (UID: \"8e99f314-66d5-4684-aec9-fe6b5c1b2e9d\") " pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:36.226469 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.226446    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:36.232315 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.232281    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7d7bcdc9c5-lssp7"]
Jul 09 16:17:36.235384 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.235363    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9mpn\" (UniqueName: \"kubernetes.io/projected/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d-kube-api-access-z9mpn\") pod \"authorino-5d4b67c446-45f4r\" (UID: \"8e99f314-66d5-4684-aec9-fe6b5c1b2e9d\") " pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:36.315763 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.315728    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:36.327158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.327131    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpxqm\" (UniqueName: \"kubernetes.io/projected/58d2f82c-aad1-4e54-b397-135b1ca3c124-kube-api-access-wpxqm\") pod \"authorino-7d7bcdc9c5-lssp7\" (UID: \"58d2f82c-aad1-4e54-b397-135b1ca3c124\") " pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:36.329409 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.329390    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:36.397558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.397520    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d7bcdc9c5-lssp7"]
Jul 09 16:17:36.397874 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:17:36.397847    2558 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wpxqm], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7" podUID="58d2f82c-aad1-4e54-b397-135b1ca3c124"
Jul 09 16:17:36.425566 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.425528    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5995455959-2l5fx"]
Jul 09 16:17:36.428490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.428466    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9mpn\" (UniqueName: \"kubernetes.io/projected/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d-kube-api-access-z9mpn\") pod \"8e99f314-66d5-4684-aec9-fe6b5c1b2e9d\" (UID: \"8e99f314-66d5-4684-aec9-fe6b5c1b2e9d\") "
Jul 09 16:17:36.428727 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.428704    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpxqm\" (UniqueName: \"kubernetes.io/projected/58d2f82c-aad1-4e54-b397-135b1ca3c124-kube-api-access-wpxqm\") pod \"authorino-7d7bcdc9c5-lssp7\" (UID: \"58d2f82c-aad1-4e54-b397-135b1ca3c124\") " pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:36.430824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.430787    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d-kube-api-access-z9mpn" (OuterVolumeSpecName: "kube-api-access-z9mpn") pod "8e99f314-66d5-4684-aec9-fe6b5c1b2e9d" (UID: "8e99f314-66d5-4684-aec9-fe6b5c1b2e9d"). InnerVolumeSpecName "kube-api-access-z9mpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:17:36.433486 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.433464    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.436437 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.436396    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 16:17:36.440216 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.440193    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5995455959-2l5fx"]
Jul 09 16:17:36.441759 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.441737    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpxqm\" (UniqueName: \"kubernetes.io/projected/58d2f82c-aad1-4e54-b397-135b1ca3c124-kube-api-access-wpxqm\") pod \"authorino-7d7bcdc9c5-lssp7\" (UID: \"58d2f82c-aad1-4e54-b397-135b1ca3c124\") " pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:36.530175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.530088    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9kzp\" (UniqueName: \"kubernetes.io/projected/36f31674-4c93-4abd-9001-07839db0e43c-kube-api-access-s9kzp\") pod \"authorino-5995455959-2l5fx\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") " pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.530175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.530129    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f31674-4c93-4abd-9001-07839db0e43c-tls-cert\") pod \"authorino-5995455959-2l5fx\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") " pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.530358 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.530245    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9mpn\" (UniqueName: \"kubernetes.io/projected/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d-kube-api-access-z9mpn\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:17:36.631722 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.631681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9kzp\" (UniqueName: \"kubernetes.io/projected/36f31674-4c93-4abd-9001-07839db0e43c-kube-api-access-s9kzp\") pod \"authorino-5995455959-2l5fx\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") " pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.631722 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.631722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f31674-4c93-4abd-9001-07839db0e43c-tls-cert\") pod \"authorino-5995455959-2l5fx\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") " pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.634415 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.634390    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f31674-4c93-4abd-9001-07839db0e43c-tls-cert\") pod \"authorino-5995455959-2l5fx\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") " pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.640228 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.640163    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9kzp\" (UniqueName: \"kubernetes.io/projected/36f31674-4c93-4abd-9001-07839db0e43c-kube-api-access-s9kzp\") pod \"authorino-5995455959-2l5fx\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") " pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.757371 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.757330    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:17:36.911141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.911113    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5995455959-2l5fx"]
Jul 09 16:17:36.913353 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:17:36.913319    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod36f31674_4c93_4abd_9001_07839db0e43c.slice/crio-d1ffc2e2c27cfab1d5918e08f8ff76defb604c7d9bb375326577c8685099030a WatchSource:0}: Error finding container d1ffc2e2c27cfab1d5918e08f8ff76defb604c7d9bb375326577c8685099030a: Status 404 returned error can't find the container with id d1ffc2e2c27cfab1d5918e08f8ff76defb604c7d9bb375326577c8685099030a
Jul 09 16:17:36.914736 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:36.914717    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:17:37.321461 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.321430    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-45f4r"
Jul 09 16:17:37.321461 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.321439    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5995455959-2l5fx" event={"ID":"36f31674-4c93-4abd-9001-07839db0e43c","Type":"ContainerStarted","Data":"d1ffc2e2c27cfab1d5918e08f8ff76defb604c7d9bb375326577c8685099030a"}
Jul 09 16:17:37.321798 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.321471    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:37.328431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.328378    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:37.350655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.350621    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-45f4r"]
Jul 09 16:17:37.354131 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.354099    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-45f4r"]
Jul 09 16:17:37.440209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.440099    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpxqm\" (UniqueName: \"kubernetes.io/projected/58d2f82c-aad1-4e54-b397-135b1ca3c124-kube-api-access-wpxqm\") pod \"58d2f82c-aad1-4e54-b397-135b1ca3c124\" (UID: \"58d2f82c-aad1-4e54-b397-135b1ca3c124\") "
Jul 09 16:17:37.442352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.442318    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58d2f82c-aad1-4e54-b397-135b1ca3c124-kube-api-access-wpxqm" (OuterVolumeSpecName: "kube-api-access-wpxqm") pod "58d2f82c-aad1-4e54-b397-135b1ca3c124" (UID: "58d2f82c-aad1-4e54-b397-135b1ca3c124"). InnerVolumeSpecName "kube-api-access-wpxqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:17:37.541455 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:37.541414    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpxqm\" (UniqueName: \"kubernetes.io/projected/58d2f82c-aad1-4e54-b397-135b1ca3c124-kube-api-access-wpxqm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:17:38.327868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.327834    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7d7bcdc9c5-lssp7"
Jul 09 16:17:38.328356 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.327835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5995455959-2l5fx" event={"ID":"36f31674-4c93-4abd-9001-07839db0e43c","Type":"ContainerStarted","Data":"de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04"}
Jul 09 16:17:38.342472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.342331    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5995455959-2l5fx" podStartSLOduration=1.9895352659999999 podStartE2EDuration="2.342312605s" totalImagesPullingTime="352.777339ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:17:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:17:38.342253661 +0000 UTC m=+803.724070453" watchObservedRunningTime="2026-07-09 16:17:38.342312605 +0000 UTC m=+803.724129394"
Jul 09 16:17:38.372609 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.372572    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7d7bcdc9c5-lssp7"]
Jul 09 16:17:38.375146 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.375106    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-skgln"]
Jul 09 16:17:38.375327 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.375301    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-skgln" podUID="a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" containerName="authorino" containerID="cri-o://87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b" gracePeriod=30
Jul 09 16:17:38.381559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.381476    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7d7bcdc9c5-lssp7"]
Jul 09 16:17:38.618024 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.617993    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:38.650426 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.650396    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4jwf\" (UniqueName: \"kubernetes.io/projected/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f-kube-api-access-c4jwf\") pod \"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f\" (UID: \"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f\") "
Jul 09 16:17:38.652796 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.652766    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f-kube-api-access-c4jwf" (OuterVolumeSpecName: "kube-api-access-c4jwf") pod "a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" (UID: "a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f"). InnerVolumeSpecName "kube-api-access-c4jwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:17:38.751229 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:38.751196    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4jwf\" (UniqueName: \"kubernetes.io/projected/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f-kube-api-access-c4jwf\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:17:39.178181 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.178147    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58d2f82c-aad1-4e54-b397-135b1ca3c124" path="/var/lib/kubelet/pods/58d2f82c-aad1-4e54-b397-135b1ca3c124/volumes"
Jul 09 16:17:39.178416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.178402    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e99f314-66d5-4684-aec9-fe6b5c1b2e9d" path="/var/lib/kubelet/pods/8e99f314-66d5-4684-aec9-fe6b5c1b2e9d/volumes"
Jul 09 16:17:39.334599 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.334564    2558 generic.go:356] "Generic (PLEG): container finished" podID="a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" containerID="87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b" exitCode=0
Jul 09 16:17:39.335108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.334630    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-skgln"
Jul 09 16:17:39.335108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.334662    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-skgln" event={"ID":"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f","Type":"ContainerDied","Data":"87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b"}
Jul 09 16:17:39.335108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.334713    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-skgln" event={"ID":"a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f","Type":"ContainerDied","Data":"18d62c03696a441e7608b2222693d86a52d6d760e4d7d6fbda5bdca9c9ce23e8"}
Jul 09 16:17:39.335108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.334737    2558 scope.go:117] "RemoveContainer" containerID="87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b"
Jul 09 16:17:39.343662 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.343638    2558 scope.go:117] "RemoveContainer" containerID="87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b"
Jul 09 16:17:39.343972 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:17:39.343950    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b\": container with ID starting with 87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b not found: ID does not exist" containerID="87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b"
Jul 09 16:17:39.344042 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.343981    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b"} err="failed to get container status \"87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b\": rpc error: code = NotFound desc = could not find container \"87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b\": container with ID starting with 87c1d13b4c002fdc9c61d6ec86deb221e7f2e7ca187abcd57bd6203ecc4ed28b not found: ID does not exist"
Jul 09 16:17:39.352401 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.352361    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-skgln"]
Jul 09 16:17:39.356195 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:39.356170    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-skgln"]
Jul 09 16:17:41.177433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:17:41.177395    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" path="/var/lib/kubelet/pods/a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f/volumes"
Jul 09 16:18:26.584581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:26.584541    2558 generic.go:356] "Generic (PLEG): container finished" podID="7851200e-a8e4-4934-8d53-0d483d116ec2" containerID="bf9841c45c0a24ff4c9326a99c84a3fe153c1b3b21a7428eae51160e20fbd82b" exitCode=0
Jul 09 16:18:26.584968 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:26.584614    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" event={"ID":"7851200e-a8e4-4934-8d53-0d483d116ec2","Type":"ContainerDied","Data":"bf9841c45c0a24ff4c9326a99c84a3fe153c1b3b21a7428eae51160e20fbd82b"}
Jul 09 16:18:26.585011 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:26.584990    2558 scope.go:117] "RemoveContainer" containerID="bf9841c45c0a24ff4c9326a99c84a3fe153c1b3b21a7428eae51160e20fbd82b"
Jul 09 16:18:26.586007 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:26.585968    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:18:26.586140 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:26.586052    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:18:26.586140 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:26.586103    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:18:27.591490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:27.591455    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" event={"ID":"7851200e-a8e4-4934-8d53-0d483d116ec2","Type":"ContainerStarted","Data":"6349db0e9aad842d119bdff671467e4020b23ff26d2df707db3e7874a8380d7d"}
Jul 09 16:18:28.230996 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:28.230957    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:18:28.231914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:28.231867    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nls8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 09 16:18:28.232028 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:28.231938    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" podUID="7851200e-a8e4-4934-8d53-0d483d116ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 09 16:18:29.231298 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:29.231258    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nls8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 09 16:18:29.231668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:29.231337    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" podUID="7851200e-a8e4-4934-8d53-0d483d116ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 09 16:18:30.231338 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:30.231291    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nls8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 09 16:18:30.231786 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:30.231363    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" podUID="7851200e-a8e4-4934-8d53-0d483d116ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 09 16:18:31.231238 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:31.231191    2558 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-nls8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Jul 09 16:18:31.231421 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:31.231271    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r" podUID="7851200e-a8e4-4934-8d53-0d483d116ec2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Jul 09 16:18:32.130706 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:32.130669    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:18:32.235015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:32.234987    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:18:32.619063 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:32.619038    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-nls8r"
Jul 09 16:18:37.981158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.981121    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"]
Jul 09 16:18:37.981820 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.981797    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" containerName="authorino"
Jul 09 16:18:37.981935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.981822    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" containerName="authorino"
Jul 09 16:18:37.982032 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.982017    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a27af3c4-0bc9-4c7c-9135-7bf977b0ee9f" containerName="authorino"
Jul 09 16:18:37.984830 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.984804    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:37.987692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.987672    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 16:18:37.987821 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.987674    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 16:18:37.989265 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.989204    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2w89m\""
Jul 09 16:18:37.990108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.989744    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 09 16:18:37.993501 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:37.993476    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"]
Jul 09 16:18:38.061251 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.061228    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvq7f\" (UniqueName: \"kubernetes.io/projected/702fdce0-61ce-4b26-b14c-c35a3002b250-kube-api-access-pvq7f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.061393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.061289    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.061393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.061308    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.061393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.061362    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.061393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.061389    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702fdce0-61ce-4b26-b14c-c35a3002b250-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.061535 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.061410    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162223    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162410 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162254    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162410 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162410 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162345    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702fdce0-61ce-4b26-b14c-c35a3002b250-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162410 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162624 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162429    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvq7f\" (UniqueName: \"kubernetes.io/projected/702fdce0-61ce-4b26-b14c-c35a3002b250-kube-api-access-pvq7f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162680 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162641    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162759 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.162817 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.162789    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.164819 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.164793    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/702fdce0-61ce-4b26-b14c-c35a3002b250-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.164952 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.164854    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/702fdce0-61ce-4b26-b14c-c35a3002b250-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.171095 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.171072    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvq7f\" (UniqueName: \"kubernetes.io/projected/702fdce0-61ce-4b26-b14c-c35a3002b250-kube-api-access-pvq7f\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr\" (UID: \"702fdce0-61ce-4b26-b14c-c35a3002b250\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.296908 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.296828    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:38.437191 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.437167    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"]
Jul 09 16:18:38.439090 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:18:38.439059    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod702fdce0_61ce_4b26_b14c_c35a3002b250.slice/crio-4ac8f38f240ce10ced7076c6b60fce670dac70b5ddc1e34671e4f7b9f3a11bc2 WatchSource:0}: Error finding container 4ac8f38f240ce10ced7076c6b60fce670dac70b5ddc1e34671e4f7b9f3a11bc2: Status 404 returned error can't find the container with id 4ac8f38f240ce10ced7076c6b60fce670dac70b5ddc1e34671e4f7b9f3a11bc2
Jul 09 16:18:38.649309 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:38.649265    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr" event={"ID":"702fdce0-61ce-4b26-b14c-c35a3002b250","Type":"ContainerStarted","Data":"4ac8f38f240ce10ced7076c6b60fce670dac70b5ddc1e34671e4f7b9f3a11bc2"}
Jul 09 16:18:40.662169 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:40.662127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr" event={"ID":"702fdce0-61ce-4b26-b14c-c35a3002b250","Type":"ContainerStarted","Data":"89b43d26a2471b6a1bfd6e83057ba5c29a628d8566b11cd1e9b1ac7de47f49af"}
Jul 09 16:18:40.680573 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:40.680514    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr" podStartSLOduration=2.474162588 podStartE2EDuration="3.680498828s" totalImagesPullingTime="1.20633624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:18:40.679313362 +0000 UTC m=+866.061130150" watchObservedRunningTime="2026-07-09 16:18:40.680498828 +0000 UTC m=+866.062315664"
Jul 09 16:18:48.296969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:48.296928    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:48.297599 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:48.297000    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:48.309632 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:48.309601    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:18:48.718988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:18:48.718954    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr"
Jul 09 16:19:15.050935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:19:15.050824    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:19:15.052368 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:19:15.052346    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:20:30.791032 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.790994    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-d6bc4c8c4-6q656"]
Jul 09 16:20:30.793494 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.793459    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:30.801569 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.801544    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d6bc4c8c4-6q656"]
Jul 09 16:20:30.847829 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.847783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d330e9e-97ce-44bf-b028-f0d4eec96434-tls-cert\") pod \"authorino-d6bc4c8c4-6q656\" (UID: \"5d330e9e-97ce-44bf-b028-f0d4eec96434\") " pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:30.848037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.847865    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhb59\" (UniqueName: \"kubernetes.io/projected/5d330e9e-97ce-44bf-b028-f0d4eec96434-kube-api-access-qhb59\") pod \"authorino-d6bc4c8c4-6q656\" (UID: \"5d330e9e-97ce-44bf-b028-f0d4eec96434\") " pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:30.948615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.948573    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhb59\" (UniqueName: \"kubernetes.io/projected/5d330e9e-97ce-44bf-b028-f0d4eec96434-kube-api-access-qhb59\") pod \"authorino-d6bc4c8c4-6q656\" (UID: \"5d330e9e-97ce-44bf-b028-f0d4eec96434\") " pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:30.948798 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.948643    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d330e9e-97ce-44bf-b028-f0d4eec96434-tls-cert\") pod \"authorino-d6bc4c8c4-6q656\" (UID: \"5d330e9e-97ce-44bf-b028-f0d4eec96434\") " pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:30.951236 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.951212    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d330e9e-97ce-44bf-b028-f0d4eec96434-tls-cert\") pod \"authorino-d6bc4c8c4-6q656\" (UID: \"5d330e9e-97ce-44bf-b028-f0d4eec96434\") " pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:30.956952 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:30.956913    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhb59\" (UniqueName: \"kubernetes.io/projected/5d330e9e-97ce-44bf-b028-f0d4eec96434-kube-api-access-qhb59\") pod \"authorino-d6bc4c8c4-6q656\" (UID: \"5d330e9e-97ce-44bf-b028-f0d4eec96434\") " pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:31.103744 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:31.103708    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-d6bc4c8c4-6q656"
Jul 09 16:20:31.263056 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:31.263028    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-d6bc4c8c4-6q656"]
Jul 09 16:20:31.265328 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:20:31.265295    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d330e9e_97ce_44bf_b028_f0d4eec96434.slice/crio-fc7eb2cba2331efe80d865747119a101c3fcb76d1cb2bebec6360fe3aa328f3c WatchSource:0}: Error finding container fc7eb2cba2331efe80d865747119a101c3fcb76d1cb2bebec6360fe3aa328f3c: Status 404 returned error can't find the container with id fc7eb2cba2331efe80d865747119a101c3fcb76d1cb2bebec6360fe3aa328f3c
Jul 09 16:20:32.240865 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.240823    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d6bc4c8c4-6q656" event={"ID":"5d330e9e-97ce-44bf-b028-f0d4eec96434","Type":"ContainerStarted","Data":"42e34cfa5a4fcde3eae6663b792d213917989fdf8ff7d11233f941c889726dcb"}
Jul 09 16:20:32.241378 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.240903    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-d6bc4c8c4-6q656" event={"ID":"5d330e9e-97ce-44bf-b028-f0d4eec96434","Type":"ContainerStarted","Data":"fc7eb2cba2331efe80d865747119a101c3fcb76d1cb2bebec6360fe3aa328f3c"}
Jul 09 16:20:32.266541 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.266477    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-d6bc4c8c4-6q656" podStartSLOduration=1.771451893 podStartE2EDuration="2.266452853s" totalImagesPullingTime="495.00096ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:20:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:20:32.265142396 +0000 UTC m=+977.646959185" watchObservedRunningTime="2026-07-09 16:20:32.266452853 +0000 UTC m=+977.648269642"
Jul 09 16:20:32.316366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.316306    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5995455959-2l5fx"]
Jul 09 16:20:32.317454 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.317161    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5995455959-2l5fx" podUID="36f31674-4c93-4abd-9001-07839db0e43c" containerName="authorino" containerID="cri-o://de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04" gracePeriod=30
Jul 09 16:20:32.590230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.590203    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:20:32.665970 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.665932    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9kzp\" (UniqueName: \"kubernetes.io/projected/36f31674-4c93-4abd-9001-07839db0e43c-kube-api-access-s9kzp\") pod \"36f31674-4c93-4abd-9001-07839db0e43c\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") "
Jul 09 16:20:32.666177 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.666048    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f31674-4c93-4abd-9001-07839db0e43c-tls-cert\") pod \"36f31674-4c93-4abd-9001-07839db0e43c\" (UID: \"36f31674-4c93-4abd-9001-07839db0e43c\") "
Jul 09 16:20:32.668401 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.668354    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f31674-4c93-4abd-9001-07839db0e43c-kube-api-access-s9kzp" (OuterVolumeSpecName: "kube-api-access-s9kzp") pod "36f31674-4c93-4abd-9001-07839db0e43c" (UID: "36f31674-4c93-4abd-9001-07839db0e43c"). InnerVolumeSpecName "kube-api-access-s9kzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:20:32.677606 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.677572    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f31674-4c93-4abd-9001-07839db0e43c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "36f31674-4c93-4abd-9001-07839db0e43c" (UID: "36f31674-4c93-4abd-9001-07839db0e43c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:20:32.767444 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.767349    2558 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/36f31674-4c93-4abd-9001-07839db0e43c-tls-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:20:32.767444 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:32.767386    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9kzp\" (UniqueName: \"kubernetes.io/projected/36f31674-4c93-4abd-9001-07839db0e43c-kube-api-access-s9kzp\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:20:33.248141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.248100    2558 generic.go:356] "Generic (PLEG): container finished" podID="36f31674-4c93-4abd-9001-07839db0e43c" containerID="de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04" exitCode=0
Jul 09 16:20:33.248581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.248153    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5995455959-2l5fx"
Jul 09 16:20:33.248581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.248187    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5995455959-2l5fx" event={"ID":"36f31674-4c93-4abd-9001-07839db0e43c","Type":"ContainerDied","Data":"de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04"}
Jul 09 16:20:33.248581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.248236    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5995455959-2l5fx" event={"ID":"36f31674-4c93-4abd-9001-07839db0e43c","Type":"ContainerDied","Data":"d1ffc2e2c27cfab1d5918e08f8ff76defb604c7d9bb375326577c8685099030a"}
Jul 09 16:20:33.248581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.248258    2558 scope.go:117] "RemoveContainer" containerID="de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04"
Jul 09 16:20:33.259841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.259813    2558 scope.go:117] "RemoveContainer" containerID="de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04"
Jul 09 16:20:33.260242 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:20:33.260213    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04\": container with ID starting with de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04 not found: ID does not exist" containerID="de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04"
Jul 09 16:20:33.260308 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.260254    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04"} err="failed to get container status \"de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04\": rpc error: code = NotFound desc = could not find container \"de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04\": container with ID starting with de52c0aa11664513ec9040fddf8d210a8c8c002b3ae1f03b0af426a63e867c04 not found: ID does not exist"
Jul 09 16:20:33.266170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.266135    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5995455959-2l5fx"]
Jul 09 16:20:33.268412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:33.268380    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5995455959-2l5fx"]
Jul 09 16:20:35.179179 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:20:35.179073    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36f31674-4c93-4abd-9001-07839db0e43c" path="/var/lib/kubelet/pods/36f31674-4c93-4abd-9001-07839db0e43c/volumes"
Jul 09 16:24:15.086913 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:24:15.086868    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:24:15.088490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:24:15.087169    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:26:41.679815 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:41.679725    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"]
Jul 09 16:26:41.680406 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:41.680026    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4" podUID="3db92a6f-2e9d-4dae-8a4c-8535dddf152d" containerName="manager" containerID="cri-o://7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05" gracePeriod=10
Jul 09 16:26:44.940171 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:44.940149    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:26:45.091949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.091923    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-extensions-socket-volume\") pod \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") "
Jul 09 16:26:45.092095 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.091983    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wcwct\" (UniqueName: \"kubernetes.io/projected/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-kube-api-access-wcwct\") pod \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\" (UID: \"3db92a6f-2e9d-4dae-8a4c-8535dddf152d\") "
Jul 09 16:26:45.092335 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.092307    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3db92a6f-2e9d-4dae-8a4c-8535dddf152d" (UID: "3db92a6f-2e9d-4dae-8a4c-8535dddf152d"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:26:45.094117 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.094094    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-kube-api-access-wcwct" (OuterVolumeSpecName: "kube-api-access-wcwct") pod "3db92a6f-2e9d-4dae-8a4c-8535dddf152d" (UID: "3db92a6f-2e9d-4dae-8a4c-8535dddf152d"). InnerVolumeSpecName "kube-api-access-wcwct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:26:45.193339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.193316    2558 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-extensions-socket-volume\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:26:45.193339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.193339    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcwct\" (UniqueName: \"kubernetes.io/projected/3db92a6f-2e9d-4dae-8a4c-8535dddf152d-kube-api-access-wcwct\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:26:45.256183 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.256154    2558 generic.go:356] "Generic (PLEG): container finished" podID="3db92a6f-2e9d-4dae-8a4c-8535dddf152d" containerID="7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05" exitCode=0
Jul 09 16:26:45.256304 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.256204    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"
Jul 09 16:26:45.256304 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.256237    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4" event={"ID":"3db92a6f-2e9d-4dae-8a4c-8535dddf152d","Type":"ContainerDied","Data":"7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05"}
Jul 09 16:26:45.256304 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.256286    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4" event={"ID":"3db92a6f-2e9d-4dae-8a4c-8535dddf152d","Type":"ContainerDied","Data":"bb75db850811e01de6b588e581a78d774fffa07600391c8760d4c20d6487c391"}
Jul 09 16:26:45.256459 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.256308    2558 scope.go:117] "RemoveContainer" containerID="7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05"
Jul 09 16:26:45.267298 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.267273    2558 scope.go:117] "RemoveContainer" containerID="7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05"
Jul 09 16:26:45.267657 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:26:45.267616    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05\": container with ID starting with 7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05 not found: ID does not exist" containerID="7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05"
Jul 09 16:26:45.267699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.267650    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05"} err="failed to get container status \"7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05\": rpc error: code = NotFound desc = could not find container \"7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05\": container with ID starting with 7584bfdc767748005cc7fc9a035a0e4b1b48ae3ea900b9a28cecd6a7a6f91a05 not found: ID does not exist"
Jul 09 16:26:45.277439 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.277412    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"]
Jul 09 16:26:45.281777 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:45.281748    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jfjc4"]
Jul 09 16:26:47.178135 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:26:47.178100    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3db92a6f-2e9d-4dae-8a4c-8535dddf152d" path="/var/lib/kubelet/pods/3db92a6f-2e9d-4dae-8a4c-8535dddf152d/volumes"
Jul 09 16:27:42.810139 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810093    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"]
Jul 09 16:27:42.810558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810466    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36f31674-4c93-4abd-9001-07839db0e43c" containerName="authorino"
Jul 09 16:27:42.810558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810477    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="36f31674-4c93-4abd-9001-07839db0e43c" containerName="authorino"
Jul 09 16:27:42.810558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810501    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3db92a6f-2e9d-4dae-8a4c-8535dddf152d" containerName="manager"
Jul 09 16:27:42.810558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810508    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3db92a6f-2e9d-4dae-8a4c-8535dddf152d" containerName="manager"
Jul 09 16:27:42.810558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810558    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="36f31674-4c93-4abd-9001-07839db0e43c" containerName="authorino"
Jul 09 16:27:42.810735 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.810568    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3db92a6f-2e9d-4dae-8a4c-8535dddf152d" containerName="manager"
Jul 09 16:27:42.812543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.812520    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:42.815549 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.815530    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cvhzc\""
Jul 09 16:27:42.824047 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.824003    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"]
Jul 09 16:27:42.995405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.995361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mljq\" (UniqueName: \"kubernetes.io/projected/f860d58f-fe0e-4199-82a3-3ad1523707e0-kube-api-access-2mljq\") pod \"kuadrant-operator-controller-manager-86878c769f-9dtbg\" (UID: \"f860d58f-fe0e-4199-82a3-3ad1523707e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:42.995586 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:42.995417    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f860d58f-fe0e-4199-82a3-3ad1523707e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9dtbg\" (UID: \"f860d58f-fe0e-4199-82a3-3ad1523707e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.096296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.096256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mljq\" (UniqueName: \"kubernetes.io/projected/f860d58f-fe0e-4199-82a3-3ad1523707e0-kube-api-access-2mljq\") pod \"kuadrant-operator-controller-manager-86878c769f-9dtbg\" (UID: \"f860d58f-fe0e-4199-82a3-3ad1523707e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.096468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.096311    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f860d58f-fe0e-4199-82a3-3ad1523707e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9dtbg\" (UID: \"f860d58f-fe0e-4199-82a3-3ad1523707e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.096677 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.096661    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f860d58f-fe0e-4199-82a3-3ad1523707e0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-9dtbg\" (UID: \"f860d58f-fe0e-4199-82a3-3ad1523707e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.104931 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.104869    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mljq\" (UniqueName: \"kubernetes.io/projected/f860d58f-fe0e-4199-82a3-3ad1523707e0-kube-api-access-2mljq\") pod \"kuadrant-operator-controller-manager-86878c769f-9dtbg\" (UID: \"f860d58f-fe0e-4199-82a3-3ad1523707e0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.123786 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.123749    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.275156 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.275123    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"]
Jul 09 16:27:43.278677 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:27:43.278647    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf860d58f_fe0e_4199_82a3_3ad1523707e0.slice/crio-ebe368344b93b8db143d0db112cc07efbaa8baeb5b1257de569058bcb394c768 WatchSource:0}: Error finding container ebe368344b93b8db143d0db112cc07efbaa8baeb5b1257de569058bcb394c768: Status 404 returned error can't find the container with id ebe368344b93b8db143d0db112cc07efbaa8baeb5b1257de569058bcb394c768
Jul 09 16:27:43.556080 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.555975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg" event={"ID":"f860d58f-fe0e-4199-82a3-3ad1523707e0","Type":"ContainerStarted","Data":"c1b86d0de6bd9eed1b4edc429f7ea0811261b4a2e5c280a6f3d7202344f3e6d4"}
Jul 09 16:27:43.556080 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.556038    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg" event={"ID":"f860d58f-fe0e-4199-82a3-3ad1523707e0","Type":"ContainerStarted","Data":"ebe368344b93b8db143d0db112cc07efbaa8baeb5b1257de569058bcb394c768"}
Jul 09 16:27:43.556282 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.556175    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:27:43.576600 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:43.576539    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg" podStartSLOduration=1.576521258 podStartE2EDuration="1.576521258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:27:43.574423251 +0000 UTC m=+1408.956240040" watchObservedRunningTime="2026-07-09 16:27:43.576521258 +0000 UTC m=+1408.958338045"
Jul 09 16:27:54.565218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:27:54.565179    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-9dtbg"
Jul 09 16:29:15.120450 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:29:15.120321    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:29:15.123762 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:29:15.120964    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:30:00.139453 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.139409    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"]
Jul 09 16:30:00.142928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.142908    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:30:00.145735 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.145709    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:30:00.145913 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.145888    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:30:00.146038 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.145910    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:30:00.157185 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.157155    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"]
Jul 09 16:30:00.271013 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.270970    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvmdd\" (UniqueName: \"kubernetes.io/projected/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1-kube-api-access-rvmdd\") pod \"maas-api-key-cleanup-29726910-4k6nt\" (UID: \"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:30:00.372078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.372030    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvmdd\" (UniqueName: \"kubernetes.io/projected/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1-kube-api-access-rvmdd\") pod \"maas-api-key-cleanup-29726910-4k6nt\" (UID: \"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:30:00.381725 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.381688    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvmdd\" (UniqueName: \"kubernetes.io/projected/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1-kube-api-access-rvmdd\") pod \"maas-api-key-cleanup-29726910-4k6nt\" (UID: \"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:30:00.452843 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.452745    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:30:00.595823 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.595791    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"]
Jul 09 16:30:00.597969 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:30:00.597934    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadb6ffb_8bfe_40f3_9729_b10ab06f17a1.slice/crio-30be6ba8a760389810ac37c1ba577ffe3e4be77675c94aff24203554c317f956 WatchSource:0}: Error finding container 30be6ba8a760389810ac37c1ba577ffe3e4be77675c94aff24203554c317f956: Status 404 returned error can't find the container with id 30be6ba8a760389810ac37c1ba577ffe3e4be77675c94aff24203554c317f956
Jul 09 16:30:00.600298 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:00.600279    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:30:01.308871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:01.308835    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt" event={"ID":"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1","Type":"ContainerStarted","Data":"30be6ba8a760389810ac37c1ba577ffe3e4be77675c94aff24203554c317f956"}
Jul 09 16:30:03.321831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:03.321786    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt" event={"ID":"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1","Type":"ContainerStarted","Data":"85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021"}
Jul 09 16:30:03.339416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:30:03.339357    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt" podStartSLOduration=0.96191658 podStartE2EDuration="3.339336108s" totalImagesPullingTime="2.377419528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:30:03.337279815 +0000 UTC m=+1548.719096606" watchObservedRunningTime="2026-07-09 16:30:03.339336108 +0000 UTC m=+1548.721152895"
Jul 09 16:31:00.010350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:00.010308    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"]
Jul 09 16:31:00.010957 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:00.010521    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt" podUID="aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" containerName="cleanup" containerID="cri-o://85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021" gracePeriod=30
Jul 09 16:31:30.201000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.200968    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:31:30.291208 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.291111    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvmdd\" (UniqueName: \"kubernetes.io/projected/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1-kube-api-access-rvmdd\") pod \"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1\" (UID: \"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1\") "
Jul 09 16:31:30.293500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.293469    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1-kube-api-access-rvmdd" (OuterVolumeSpecName: "kube-api-access-rvmdd") pod "aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" (UID: "aadb6ffb-8bfe-40f3-9729-b10ab06f17a1"). InnerVolumeSpecName "kube-api-access-rvmdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:31:30.392608 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.392553    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvmdd\" (UniqueName: \"kubernetes.io/projected/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1-kube-api-access-rvmdd\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:31:30.802057 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.802019    2558 generic.go:356] "Generic (PLEG): container finished" podID="aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" containerID="85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021" exitCode=137
Jul 09 16:31:30.802057 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.802059    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"
Jul 09 16:31:30.802370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.802078    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt" event={"ID":"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1","Type":"ContainerDied","Data":"85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021"}
Jul 09 16:31:30.802370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.802130    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt" event={"ID":"aadb6ffb-8bfe-40f3-9729-b10ab06f17a1","Type":"ContainerDied","Data":"30be6ba8a760389810ac37c1ba577ffe3e4be77675c94aff24203554c317f956"}
Jul 09 16:31:30.802370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.802154    2558 scope.go:117] "RemoveContainer" containerID="85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021"
Jul 09 16:31:30.811984 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.811962    2558 scope.go:117] "RemoveContainer" containerID="85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021"
Jul 09 16:31:30.812324 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:31:30.812302    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021\": container with ID starting with 85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021 not found: ID does not exist" containerID="85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021"
Jul 09 16:31:30.812405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.812333    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021"} err="failed to get container status \"85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021\": rpc error: code = NotFound desc = could not find container \"85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021\": container with ID starting with 85d2a204b657e918692e5f38dca195ba02d1ce24931c1236a7b7929e96483021 not found: ID does not exist"
Jul 09 16:31:30.823576 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.823540    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"]
Jul 09 16:31:30.826732 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:30.826702    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29726910-4k6nt"]
Jul 09 16:31:31.178284 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:31:31.178247    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" path="/var/lib/kubelet/pods/aadb6ffb-8bfe-40f3-9729-b10ab06f17a1/volumes"
Jul 09 16:34:05.784386 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.784298    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"]
Jul 09 16:34:05.784858 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.784837    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" containerName="cleanup"
Jul 09 16:34:05.784931 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.784863    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" containerName="cleanup"
Jul 09 16:34:05.785014 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.785000    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadb6ffb-8bfe-40f3-9729-b10ab06f17a1" containerName="cleanup"
Jul 09 16:34:05.788420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.788398    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.791374 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.791356    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-4dabc427-openshift-default-dockercfg-c2wgc\""
Jul 09 16:34:05.803935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.803906    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"]
Jul 09 16:34:05.895968 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.895927    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-certs\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.895971    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nfbt\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-kube-api-access-9nfbt\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896079    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-data\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896114    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-podinfo\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896202    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-token\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896244    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-credential-socket\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896385 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896316    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-socket\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896385 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896371    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-envoy\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.896457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.896400    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istiod-ca-cert\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997452 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-socket\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997452 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997456    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-envoy\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997684 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997576    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istiod-ca-cert\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997684 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997618    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-certs\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997684 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997642    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nfbt\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-kube-api-access-9nfbt\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997810 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997709    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-data\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997810 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997734    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-podinfo\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997810 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997785    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-token\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997820    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-credential-socket\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997856    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-socket\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.997987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.997969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-certs\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.998126 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.998067    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-data\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.998177 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.998165    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-credential-socket\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:05.998335 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:05.998312    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istiod-ca-cert\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:06.000137 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.000106    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-envoy\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:06.000303 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.000285    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-podinfo\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:06.006413 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.006386    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-token\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:06.006952 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.006932    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nfbt\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-kube-api-access-9nfbt\") pod \"e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") " pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:06.098790 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.098743    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:06.242857 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.242821    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"]
Jul 09 16:34:06.246108 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:34:06.246080    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9215cfa9_bc76_4de9_bd65_dadf869dd6ba.slice/crio-1ca43e1a3f23b331ba51bc0a1e181e16e4bedc6a510f930695a4525ce5a324da WatchSource:0}: Error finding container 1ca43e1a3f23b331ba51bc0a1e181e16e4bedc6a510f930695a4525ce5a324da: Status 404 returned error can't find the container with id 1ca43e1a3f23b331ba51bc0a1e181e16e4bedc6a510f930695a4525ce5a324da
Jul 09 16:34:06.248536 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.248503    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:06.248630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.248570    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:06.248630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.248608    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:06.655113 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.655076    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9" event={"ID":"9215cfa9-bc76-4de9-bd65-dadf869dd6ba","Type":"ContainerStarted","Data":"45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c"}
Jul 09 16:34:06.655302 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.655127    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9" event={"ID":"9215cfa9-bc76-4de9-bd65-dadf869dd6ba","Type":"ContainerStarted","Data":"1ca43e1a3f23b331ba51bc0a1e181e16e4bedc6a510f930695a4525ce5a324da"}
Jul 09 16:34:06.677963 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.677873    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9" podStartSLOduration=1.677853314 podStartE2EDuration="1.677853314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:06.675997767 +0000 UTC m=+1792.057814554" watchObservedRunningTime="2026-07-09 16:34:06.677853314 +0000 UTC m=+1792.059670100"
Jul 09 16:34:06.998381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:06.998296    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"]
Jul 09 16:34:07.008246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.008210    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.012630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.012597    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-4dabc427-serving-cert\""
Jul 09 16:34:07.013163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.012927    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:34:07.013163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.013008    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:34:07.013163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.013042    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:34:07.013416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.013337    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"]
Jul 09 16:34:07.099337 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.099283    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:07.107767 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.107718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.107969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.107782    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-openshift-service-ca\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.107969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.107811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rv7\" (UniqueName: \"kubernetes.io/projected/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-kube-api-access-r7rv7\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.114134 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.114061    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:07.209043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.209000    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.209249 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.209067    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-openshift-service-ca\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.209249 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.209094    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rv7\" (UniqueName: \"kubernetes.io/projected/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-kube-api-access-r7rv7\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.209249 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:07.209193    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-serving-cert: secret "maas-api-e2e-ait-4dabc427-serving-cert" not found
Jul 09 16:34:07.209458 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:07.209275    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls podName:a383ae76-eecd-4d3a-a7a8-0473b0c16ccc nodeName:}" failed. No retries permitted until 2026-07-09 16:34:07.709249956 +0000 UTC m=+1793.091066741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls") pod "maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" (UID: "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc") : secret "maas-api-e2e-ait-4dabc427-serving-cert" not found
Jul 09 16:34:07.210010 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.209983    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-openshift-service-ca\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.219208 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.219179    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rv7\" (UniqueName: \"kubernetes.io/projected/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-kube-api-access-r7rv7\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.661210 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.661182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:07.662158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.662137    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:07.714196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.714166    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.716909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.716865    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls\") pod \"maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:07.923277 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:07.923181    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:08.087387 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:08.087353    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"]
Jul 09 16:34:08.088627 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:34:08.088601    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda383ae76_eecd_4d3a_a7a8_0473b0c16ccc.slice/crio-12494c81334b95b014980ea732c5dbe81e62b653f2d767d81aa5364993bdca7e WatchSource:0}: Error finding container 12494c81334b95b014980ea732c5dbe81e62b653f2d767d81aa5364993bdca7e: Status 404 returned error can't find the container with id 12494c81334b95b014980ea732c5dbe81e62b653f2d767d81aa5364993bdca7e
Jul 09 16:34:08.672889 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:08.672850    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" event={"ID":"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc","Type":"ContainerStarted","Data":"12494c81334b95b014980ea732c5dbe81e62b653f2d767d81aa5364993bdca7e"}
Jul 09 16:34:10.687091 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:10.687050    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" event={"ID":"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc","Type":"ContainerStarted","Data":"a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9"}
Jul 09 16:34:10.687484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:10.687131    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:10.705189 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:10.705137    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" podStartSLOduration=2.775715499 podStartE2EDuration="4.705120485s" totalImagesPullingTime="1.929404986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:10.703338609 +0000 UTC m=+1796.085155417" watchObservedRunningTime="2026-07-09 16:34:10.705120485 +0000 UTC m=+1796.086937272"
Jul 09 16:34:12.190472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:12.190429    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"]
Jul 09 16:34:12.190977 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:12.190663    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9" podUID="9215cfa9-bc76-4de9-bd65-dadf869dd6ba" containerName="istio-proxy" containerID="cri-o://45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c" gracePeriod=30
Jul 09 16:34:12.347909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:12.345504    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"]
Jul 09 16:34:12.697312 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:12.697256    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" podUID="a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" containerName="maas-api" containerID="cri-o://a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9" gracePeriod=30
Jul 09 16:34:12.990177 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:12.990151    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:13.064212 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.064183    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls\") pod \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") "
Jul 09 16:34:13.064372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.064230    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-openshift-service-ca\") pod \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") "
Jul 09 16:34:13.064372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.064250    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7rv7\" (UniqueName: \"kubernetes.io/projected/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-kube-api-access-r7rv7\") pod \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\" (UID: \"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc\") "
Jul 09 16:34:13.064546 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.064521    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" (UID: "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:34:13.066513 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.066487    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-kube-api-access-r7rv7" (OuterVolumeSpecName: "kube-api-access-r7rv7") pod "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" (UID: "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc"). InnerVolumeSpecName "kube-api-access-r7rv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:13.066618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.066552    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" (UID: "a383ae76-eecd-4d3a-a7a8-0473b0c16ccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:34:13.165611 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.165569    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:13.165611 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.165606    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:13.165611 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.165616    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rv7\" (UniqueName: \"kubernetes.io/projected/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc-kube-api-access-r7rv7\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:13.724904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.720658    2558 generic.go:356] "Generic (PLEG): container finished" podID="a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" containerID="a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9" exitCode=0
Jul 09 16:34:13.724904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.720907    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" event={"ID":"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc","Type":"ContainerDied","Data":"a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9"}
Jul 09 16:34:13.724904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.721022    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9" event={"ID":"a383ae76-eecd-4d3a-a7a8-0473b0c16ccc","Type":"ContainerDied","Data":"12494c81334b95b014980ea732c5dbe81e62b653f2d767d81aa5364993bdca7e"}
Jul 09 16:34:13.724904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.721089    2558 scope.go:117] "RemoveContainer" containerID="a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9"
Jul 09 16:34:13.724904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.721333    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"
Jul 09 16:34:13.745915 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.745040    2558 scope.go:117] "RemoveContainer" containerID="a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9"
Jul 09 16:34:13.747682 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.747655    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"]
Jul 09 16:34:13.748195 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:13.748155    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9\": container with ID starting with a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9 not found: ID does not exist" containerID="a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9"
Jul 09 16:34:13.748290 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.748203    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9"} err="failed to get container status \"a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9\": rpc error: code = NotFound desc = could not find container \"a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9\": container with ID starting with a6aa8fa2a7484d833aada92f1862d033abea067aeda1b198b0b4d0f27bc66da9 not found: ID does not exist"
Jul 09 16:34:13.751638 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:13.751613    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-4dabc427-5c9ccb455c-mlpx9"]
Jul 09 16:34:15.162448 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:15.162331    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:34:15.180394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:15.163897    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:34:15.180394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:15.179297    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" path="/var/lib/kubelet/pods/a383ae76-eecd-4d3a-a7a8-0473b0c16ccc/volumes"
Jul 09 16:34:18.306206 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.304293    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"]
Jul 09 16:34:18.306206 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.304678    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" containerName="maas-api"
Jul 09 16:34:18.306206 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.304699    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" containerName="maas-api"
Jul 09 16:34:18.306206 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.304980    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a383ae76-eecd-4d3a-a7a8-0473b0c16ccc" containerName="maas-api"
Jul 09 16:34:18.310200 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.310121    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.314297 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.314050    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fbabafff-openshift-default-dockercfg-w5626\""
Jul 09 16:34:18.323307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.323269    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"]
Jul 09 16:34:18.356756 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.356726    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:18.415452 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.415353    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-socket\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.415968 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.415923    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.415965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istiod-ca-cert\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416010    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-data\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-token\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416103    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nfbt\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-kube-api-access-9nfbt\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416135    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-podinfo\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-certs\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416243    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-envoy\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416301    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-credential-socket\") pod \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\" (UID: \"9215cfa9-bc76-4de9-bd65-dadf869dd6ba\") "
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416514    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-token\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-certs\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416627    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a694363-1432-4dd1-94f1-f9e6232b8531-istiod-ca-cert\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416670    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-credential-socket\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416713    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-podinfo\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416751    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-envoy\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416773    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-data\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.417596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416811    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-socket\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.418668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.416851    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-kube-api-access-9dhqr\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.418668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.417200    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:18.418668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.417377    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:18.418668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.417738    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:34:18.418668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.417775    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-data" (OuterVolumeSpecName: "istio-data") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:18.420218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.420175    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:18.420866 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.420841    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:34:18.420989 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.420933    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-token" (OuterVolumeSpecName: "istio-token") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:18.421301 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.421284    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-kube-api-access-9nfbt" (OuterVolumeSpecName: "kube-api-access-9nfbt") pod "9215cfa9-bc76-4de9-bd65-dadf869dd6ba" (UID: "9215cfa9-bc76-4de9-bd65-dadf869dd6ba"). InnerVolumeSpecName "kube-api-access-9nfbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:18.518264 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518228    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-token\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518264 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518283    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-certs\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518299    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a694363-1432-4dd1-94f1-f9e6232b8531-istiod-ca-cert\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518322    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-credential-socket\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518360    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-podinfo\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518391    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-envoy\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-data\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518441    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-socket\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518475    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-kube-api-access-9dhqr\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518572    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518587    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518602    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518615    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518645    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nfbt\" (UniqueName: \"kubernetes.io/projected/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-kube-api-access-9nfbt\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518658    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518680    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518692    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518704    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9215cfa9-bc76-4de9-bd65-dadf869dd6ba-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518747    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-certs\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.518868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.518787    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-credential-socket\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.519597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.519116    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-socket\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.519597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.519120    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-data\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.519597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.519164    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a694363-1432-4dd1-94f1-f9e6232b8531-istiod-ca-cert\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.520951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.520929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-envoy\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.521155 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.521138    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-podinfo\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.527388 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.527362    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-token\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.527491 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.527477    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-kube-api-access-9dhqr\") pod \"e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") " pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.627373 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.627324    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:18.754844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.754811    2558 generic.go:356] "Generic (PLEG): container finished" podID="9215cfa9-bc76-4de9-bd65-dadf869dd6ba" containerID="45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c" exitCode=0
Jul 09 16:34:18.755025 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.754859    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9" event={"ID":"9215cfa9-bc76-4de9-bd65-dadf869dd6ba","Type":"ContainerDied","Data":"45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c"}
Jul 09 16:34:18.755025 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.754865    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"
Jul 09 16:34:18.755025 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.754930    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9" event={"ID":"9215cfa9-bc76-4de9-bd65-dadf869dd6ba","Type":"ContainerDied","Data":"1ca43e1a3f23b331ba51bc0a1e181e16e4bedc6a510f930695a4525ce5a324da"}
Jul 09 16:34:18.755025 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.754960    2558 scope.go:117] "RemoveContainer" containerID="45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c"
Jul 09 16:34:18.764548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.764531    2558 scope.go:117] "RemoveContainer" containerID="45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c"
Jul 09 16:34:18.764870 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:18.764799    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c\": container with ID starting with 45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c not found: ID does not exist" containerID="45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c"
Jul 09 16:34:18.764870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.764834    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c"} err="failed to get container status \"45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c\": rpc error: code = NotFound desc = could not find container \"45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c\": container with ID starting with 45dcc5e420eb084da42157526af4ee4f6ce445a643b367fc27a670773972520c not found: ID does not exist"
Jul 09 16:34:18.772412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.772392    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"]
Jul 09 16:34:18.774324 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:34:18.774291    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a694363_1432_4dd1_94f1_f9e6232b8531.slice/crio-de5184925d84ddfb55dd4c9ded569cb12542581214bf005450999a256e473bac WatchSource:0}: Error finding container de5184925d84ddfb55dd4c9ded569cb12542581214bf005450999a256e473bac: Status 404 returned error can't find the container with id de5184925d84ddfb55dd4c9ded569cb12542581214bf005450999a256e473bac
Jul 09 16:34:18.783074 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.783043    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"]
Jul 09 16:34:18.783764 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.783724    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:18.783863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.783828    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:18.783957 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.783872    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:18.786726 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:18.786703    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-4dabc427-openshift-default-6d59ff4bf-tq9j9"]
Jul 09 16:34:19.181221 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.181186    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9215cfa9-bc76-4de9-bd65-dadf869dd6ba" path="/var/lib/kubelet/pods/9215cfa9-bc76-4de9-bd65-dadf869dd6ba/volumes"
Jul 09 16:34:19.519218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.519137    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"]
Jul 09 16:34:19.519806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.519784    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9215cfa9-bc76-4de9-bd65-dadf869dd6ba" containerName="istio-proxy"
Jul 09 16:34:19.519925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.519808    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="9215cfa9-bc76-4de9-bd65-dadf869dd6ba" containerName="istio-proxy"
Jul 09 16:34:19.519925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.519917    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="9215cfa9-bc76-4de9-bd65-dadf869dd6ba" containerName="istio-proxy"
Jul 09 16:34:19.523135 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.523111    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.526166 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.526130    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:34:19.526294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.526218    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fbabafff-serving-cert\""
Jul 09 16:34:19.526537 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.526514    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:34:19.526771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.526747    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:34:19.534079 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.534056    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"]
Jul 09 16:34:19.627931 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.627858    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7qmh\" (UniqueName: \"kubernetes.io/projected/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-kube-api-access-k7qmh\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.628151 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.628010    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-maas-api-tls\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.628151 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.628040    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-openshift-service-ca\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.728519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.728478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-maas-api-tls\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.728519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.728518    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-openshift-service-ca\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.728841 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.728602    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7qmh\" (UniqueName: \"kubernetes.io/projected/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-kube-api-access-k7qmh\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.729262 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.729239    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-openshift-service-ca\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.731404 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.731377    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-maas-api-tls\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.738106 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.738080    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7qmh\" (UniqueName: \"kubernetes.io/projected/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-kube-api-access-k7qmh\") pod \"maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.762181 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.762143    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9" event={"ID":"5a694363-1432-4dd1-94f1-f9e6232b8531","Type":"ContainerStarted","Data":"78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae"}
Jul 09 16:34:19.762350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.762204    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9" event={"ID":"5a694363-1432-4dd1-94f1-f9e6232b8531","Type":"ContainerStarted","Data":"de5184925d84ddfb55dd4c9ded569cb12542581214bf005450999a256e473bac"}
Jul 09 16:34:19.785163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.785045    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9" podStartSLOduration=1.785028059 podStartE2EDuration="1.785028059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:19.782648972 +0000 UTC m=+1805.164465759" watchObservedRunningTime="2026-07-09 16:34:19.785028059 +0000 UTC m=+1805.166844846"
Jul 09 16:34:19.834838 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.834805    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:19.978542 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:19.978509    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"]
Jul 09 16:34:19.979198 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:34:19.979175    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ba01c7_44c0_4f3a_b15c_a30fa6ba3e87.slice/crio-43507c2f291499abc34216ff6dfdd53f21886a2fefb8502be79e5dc7fdd65984 WatchSource:0}: Error finding container 43507c2f291499abc34216ff6dfdd53f21886a2fefb8502be79e5dc7fdd65984: Status 404 returned error can't find the container with id 43507c2f291499abc34216ff6dfdd53f21886a2fefb8502be79e5dc7fdd65984
Jul 09 16:34:20.627860 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.627821    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:20.633548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.633521    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:20.770075 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.769970    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2" event={"ID":"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87","Type":"ContainerStarted","Data":"5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75"}
Jul 09 16:34:20.770247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.770128    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:20.770247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.770167    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2" event={"ID":"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87","Type":"ContainerStarted","Data":"43507c2f291499abc34216ff6dfdd53f21886a2fefb8502be79e5dc7fdd65984"}
Jul 09 16:34:20.770247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.770195    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:20.770980 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.770962    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:20.787808 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:20.787754    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2" podStartSLOduration=1.312011258 podStartE2EDuration="1.787736269s" totalImagesPullingTime="475.725011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:20.786798992 +0000 UTC m=+1806.168615778" watchObservedRunningTime="2026-07-09 16:34:20.787736269 +0000 UTC m=+1806.169553055"
Jul 09 16:34:24.850730 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:24.850690    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"]
Jul 09 16:34:24.851175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:24.850963    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2" podUID="f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" containerName="maas-api" containerID="cri-o://5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75" gracePeriod=30
Jul 09 16:34:25.145740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.145711    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:25.178891 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.178840    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7qmh\" (UniqueName: \"kubernetes.io/projected/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-kube-api-access-k7qmh\") pod \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") "
Jul 09 16:34:25.179081 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.178924    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-maas-api-tls\") pod \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") "
Jul 09 16:34:25.179081 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.179035    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-openshift-service-ca\") pod \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\" (UID: \"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87\") "
Jul 09 16:34:25.179416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.179389    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" (UID: "f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:34:25.181036 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.181018    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" (UID: "f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:34:25.181095 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.181075    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-kube-api-access-k7qmh" (OuterVolumeSpecName: "kube-api-access-k7qmh") pod "f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" (UID: "f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87"). InnerVolumeSpecName "kube-api-access-k7qmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:25.280572 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.280528    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7qmh\" (UniqueName: \"kubernetes.io/projected/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-kube-api-access-k7qmh\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:25.280572 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.280561    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:25.280572 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.280570    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:25.798585 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.798551    2558 generic.go:356] "Generic (PLEG): container finished" podID="f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" containerID="5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75" exitCode=0
Jul 09 16:34:25.798774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.798599    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"
Jul 09 16:34:25.798774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.798645    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2" event={"ID":"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87","Type":"ContainerDied","Data":"5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75"}
Jul 09 16:34:25.798774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.798698    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2" event={"ID":"f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87","Type":"ContainerDied","Data":"43507c2f291499abc34216ff6dfdd53f21886a2fefb8502be79e5dc7fdd65984"}
Jul 09 16:34:25.798774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.798720    2558 scope.go:117] "RemoveContainer" containerID="5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75"
Jul 09 16:34:25.815097 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.809220    2558 scope.go:117] "RemoveContainer" containerID="5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75"
Jul 09 16:34:25.815097 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:25.810460    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75\": container with ID starting with 5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75 not found: ID does not exist" containerID="5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75"
Jul 09 16:34:25.815097 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.810500    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75"} err="failed to get container status \"5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75\": rpc error: code = NotFound desc = could not find container \"5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75\": container with ID starting with 5f7579b8e6ead23ae78d3393731822d394621087c046725f820335e1671ccd75 not found: ID does not exist"
Jul 09 16:34:25.822314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.822268    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"]
Jul 09 16:34:25.826832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:25.826795    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fbabafff-79f48dd596-6w8w2"]
Jul 09 16:34:27.182021 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:27.181981    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" path="/var/lib/kubelet/pods/f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87/volumes"
Jul 09 16:34:30.594719 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.594688    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"]
Jul 09 16:34:30.595223 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.594898    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9" podUID="5a694363-1432-4dd1-94f1-f9e6232b8531" containerName="istio-proxy" containerID="cri-o://78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae" gracePeriod=30
Jul 09 16:34:30.749942 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.749916    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:30.830040 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.830007    2558 generic.go:356] "Generic (PLEG): container finished" podID="5a694363-1432-4dd1-94f1-f9e6232b8531" containerID="78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae" exitCode=0
Jul 09 16:34:30.830250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.830061    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"
Jul 09 16:34:30.830250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.830092    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9" event={"ID":"5a694363-1432-4dd1-94f1-f9e6232b8531","Type":"ContainerDied","Data":"78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae"}
Jul 09 16:34:30.830250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.830149    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9" event={"ID":"5a694363-1432-4dd1-94f1-f9e6232b8531","Type":"ContainerDied","Data":"de5184925d84ddfb55dd4c9ded569cb12542581214bf005450999a256e473bac"}
Jul 09 16:34:30.830250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.830174    2558 scope.go:117] "RemoveContainer" containerID="78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae"
Jul 09 16:34:30.834951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.834925    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-podinfo\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835102 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.834971    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-credential-socket\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835102 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835011    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-certs\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835346 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835316    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:30.835346 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835328    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:30.835529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835405    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-envoy\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835443    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-socket\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835472    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-token\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835497    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-kube-api-access-9dhqr\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835524    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a694363-1432-4dd1-94f1-f9e6232b8531-istiod-ca-cert\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835756 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835599    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-data\") pod \"5a694363-1432-4dd1-94f1-f9e6232b8531\" (UID: \"5a694363-1432-4dd1-94f1-f9e6232b8531\") "
Jul 09 16:34:30.835756 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835739    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:30.836076 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835969    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.836076 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.835991    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.836076 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.836006    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.836262 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.836190    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a694363-1432-4dd1-94f1-f9e6232b8531-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:34:30.836922 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.836858    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:30.837982 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.837859    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:34:30.837982 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.837946    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:30.838130 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.838115    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:30.838688 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.838637    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-kube-api-access-9dhqr" (OuterVolumeSpecName: "kube-api-access-9dhqr") pod "5a694363-1432-4dd1-94f1-f9e6232b8531" (UID: "5a694363-1432-4dd1-94f1-f9e6232b8531"). InnerVolumeSpecName "kube-api-access-9dhqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:30.840573 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.840552    2558 scope.go:117] "RemoveContainer" containerID="78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae"
Jul 09 16:34:30.840949 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:30.840921    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae\": container with ID starting with 78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae not found: ID does not exist" containerID="78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae"
Jul 09 16:34:30.841044 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.840960    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae"} err="failed to get container status \"78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae\": rpc error: code = NotFound desc = could not find container \"78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae\": container with ID starting with 78ce7f12b4afa141574219c194c15bfd38284668188d689388a15318026916ae not found: ID does not exist"
Jul 09 16:34:30.936827 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.936796    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.936827 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.936826    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.936827 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.936837    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.937115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.936849    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dhqr\" (UniqueName: \"kubernetes.io/projected/5a694363-1432-4dd1-94f1-f9e6232b8531-kube-api-access-9dhqr\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.937115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.936861    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a694363-1432-4dd1-94f1-f9e6232b8531-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:30.937115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:30.936891    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a694363-1432-4dd1-94f1-f9e6232b8531-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:31.151873 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:31.151834    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"]
Jul 09 16:34:31.155536 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:31.155508    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fbabafff-openshift-default-6c4fc4cf79-vc4s9"]
Jul 09 16:34:31.178162 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:31.178133    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a694363-1432-4dd1-94f1-f9e6232b8531" path="/var/lib/kubelet/pods/5a694363-1432-4dd1-94f1-f9e6232b8531/volumes"
Jul 09 16:34:36.794790 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.794756    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"]
Jul 09 16:34:36.795416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.795132    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a694363-1432-4dd1-94f1-f9e6232b8531" containerName="istio-proxy"
Jul 09 16:34:36.795416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.795145    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a694363-1432-4dd1-94f1-f9e6232b8531" containerName="istio-proxy"
Jul 09 16:34:36.795416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.795161    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" containerName="maas-api"
Jul 09 16:34:36.795416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.795167    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" containerName="maas-api"
Jul 09 16:34:36.795416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.795224    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ba01c7-44c0-4f3a-b15c-a30fa6ba3e87" containerName="maas-api"
Jul 09 16:34:36.795416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.795234    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a694363-1432-4dd1-94f1-f9e6232b8531" containerName="istio-proxy"
Jul 09 16:34:36.799957 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.799710    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.802911 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.802891    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-5a1c9889-openshift-default-dockercfg-gjgpc\""
Jul 09 16:34:36.815216 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.815187    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"]
Jul 09 16:34:36.883961 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.883921    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfznw\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-kube-api-access-jfznw\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884161 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.883974    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-socket\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884161 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884014    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-data\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884161 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884030    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-token\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884161 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884066    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-podinfo\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884161 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884141    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-credential-socket\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884335 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884172    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-envoy\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884335 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884203    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-certs\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.884335 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.884237    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6150320-0ea0-432e-990a-a3d62791b8d0-istiod-ca-cert\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.984994 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.984956    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-podinfo\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-credential-socket\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985123    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-envoy\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-certs\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985202    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6150320-0ea0-432e-990a-a3d62791b8d0-istiod-ca-cert\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985258    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfznw\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-kube-api-access-jfznw\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-socket\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985343    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-data\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985368    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-token\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985629 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985419    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-credential-socket\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985629 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985465    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-certs\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985737 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985662    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-data\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985737 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985714    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-socket\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.985924 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.985904    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6150320-0ea0-432e-990a-a3d62791b8d0-istiod-ca-cert\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.987479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.987462    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-envoy\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:36.987637 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:36.987620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-podinfo\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:37.000453 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.000423    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-token\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:37.000620 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.000600    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfznw\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-kube-api-access-jfznw\") pod \"e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") " pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:37.111244 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.111200    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:37.257204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.257177    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"]
Jul 09 16:34:37.258376 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:34:37.258345    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6150320_0ea0_432e_990a_a3d62791b8d0.slice/crio-43d8ea878f28a9341f8da65a72c9d6e01632b02f4c898f75381054e10d4a82e1 WatchSource:0}: Error finding container 43d8ea878f28a9341f8da65a72c9d6e01632b02f4c898f75381054e10d4a82e1: Status 404 returned error can't find the container with id 43d8ea878f28a9341f8da65a72c9d6e01632b02f4c898f75381054e10d4a82e1
Jul 09 16:34:37.260394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.260354    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:37.260502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.260448    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:37.260502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.260493    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:34:37.876405 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.876371    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb" event={"ID":"f6150320-0ea0-432e-990a-a3d62791b8d0","Type":"ContainerStarted","Data":"ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068"}
Jul 09 16:34:37.876836 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.876425    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb" event={"ID":"f6150320-0ea0-432e-990a-a3d62791b8d0","Type":"ContainerStarted","Data":"43d8ea878f28a9341f8da65a72c9d6e01632b02f4c898f75381054e10d4a82e1"}
Jul 09 16:34:37.916964 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.916719    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb" podStartSLOduration=1.9166951110000001 podStartE2EDuration="1.916695111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:37.907025758 +0000 UTC m=+1823.288842544" watchObservedRunningTime="2026-07-09 16:34:37.916695111 +0000 UTC m=+1823.298511898"
Jul 09 16:34:37.994655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.994619    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"]
Jul 09 16:34:37.998323 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:37.998299    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.001234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.001202    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-5a1c9889-serving-cert\""
Jul 09 16:34:38.001522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.001246    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:34:38.001640 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.001367    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:34:38.001640 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.001411    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:34:38.011592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.011564    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"]
Jul 09 16:34:38.098111 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.098068    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7e931-85a3-4570-97d4-6828d7ac56a6-openshift-service-ca\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.098111 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.098170    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.098111 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.098233    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczw6\" (UniqueName: \"kubernetes.io/projected/13d7e931-85a3-4570-97d4-6828d7ac56a6-kube-api-access-lczw6\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.112278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.112182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:38.124205 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.124175    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:38.199708 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.199612    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.199708 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.199671    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczw6\" (UniqueName: \"kubernetes.io/projected/13d7e931-85a3-4570-97d4-6828d7ac56a6-kube-api-access-lczw6\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.199994 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.199744    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7e931-85a3-4570-97d4-6828d7ac56a6-openshift-service-ca\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.199994 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:38.199774    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-serving-cert: secret "maas-api-e2e-derive-5a1c9889-serving-cert" not found
Jul 09 16:34:38.199994 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:38.199847    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls podName:13d7e931-85a3-4570-97d4-6828d7ac56a6 nodeName:}" failed. No retries permitted until 2026-07-09 16:34:38.699823426 +0000 UTC m=+1824.081640203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls") pod "maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" (UID: "13d7e931-85a3-4570-97d4-6828d7ac56a6") : secret "maas-api-e2e-derive-5a1c9889-serving-cert" not found
Jul 09 16:34:38.200439 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.200417    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7e931-85a3-4570-97d4-6828d7ac56a6-openshift-service-ca\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.209142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.209113    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczw6\" (UniqueName: \"kubernetes.io/projected/13d7e931-85a3-4570-97d4-6828d7ac56a6-kube-api-access-lczw6\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.703731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.703681    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.706252 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.706227    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls\") pod \"maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:38.882290 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.882258    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:38.883171 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.883151    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:38.910655 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:38.910619    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:39.085808 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:39.085780    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"]
Jul 09 16:34:39.087947 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:34:39.087912    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d7e931_85a3_4570_97d4_6828d7ac56a6.slice/crio-a8c4428356dc41985c832036e6cd9f997b02b8720a781ecbe23dcd0e8aca4a95 WatchSource:0}: Error finding container a8c4428356dc41985c832036e6cd9f997b02b8720a781ecbe23dcd0e8aca4a95: Status 404 returned error can't find the container with id a8c4428356dc41985c832036e6cd9f997b02b8720a781ecbe23dcd0e8aca4a95
Jul 09 16:34:39.888416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:39.888376    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" event={"ID":"13d7e931-85a3-4570-97d4-6828d7ac56a6","Type":"ContainerStarted","Data":"342832a30aaa5840806a55ac132b960c10950e04f6a9cf839e80b25978caf1eb"}
Jul 09 16:34:39.888835 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:39.888435    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" event={"ID":"13d7e931-85a3-4570-97d4-6828d7ac56a6","Type":"ContainerStarted","Data":"a8c4428356dc41985c832036e6cd9f997b02b8720a781ecbe23dcd0e8aca4a95"}
Jul 09 16:34:39.888835 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:39.888519    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:39.907167 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:39.907114    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" podStartSLOduration=2.506342489 podStartE2EDuration="2.90709842s" totalImagesPullingTime="400.755931ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:34:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:34:39.905421028 +0000 UTC m=+1825.287237815" watchObservedRunningTime="2026-07-09 16:34:39.90709842 +0000 UTC m=+1825.288915207"
Jul 09 16:34:42.519591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.519555    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"]
Jul 09 16:34:42.520055 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.519751    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb" podUID="f6150320-0ea0-432e-990a-a3d62791b8d0" containerName="istio-proxy" containerID="cri-o://ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068" gracePeriod=30
Jul 09 16:34:42.683338 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.683306    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"]
Jul 09 16:34:42.683536 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.683509    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" podUID="13d7e931-85a3-4570-97d4-6828d7ac56a6" containerName="maas-api" containerID="cri-o://342832a30aaa5840806a55ac132b960c10950e04f6a9cf839e80b25978caf1eb" gracePeriod=30
Jul 09 16:34:42.907909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.907856    2558 generic.go:356] "Generic (PLEG): container finished" podID="13d7e931-85a3-4570-97d4-6828d7ac56a6" containerID="342832a30aaa5840806a55ac132b960c10950e04f6a9cf839e80b25978caf1eb" exitCode=0
Jul 09 16:34:42.908074 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.907934    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" event={"ID":"13d7e931-85a3-4570-97d4-6828d7ac56a6","Type":"ContainerDied","Data":"342832a30aaa5840806a55ac132b960c10950e04f6a9cf839e80b25978caf1eb"}
Jul 09 16:34:42.929651 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:42.929626    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:43.042366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.042330    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7e931-85a3-4570-97d4-6828d7ac56a6-openshift-service-ca\") pod \"13d7e931-85a3-4570-97d4-6828d7ac56a6\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") "
Jul 09 16:34:43.042554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.042404    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls\") pod \"13d7e931-85a3-4570-97d4-6828d7ac56a6\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") "
Jul 09 16:34:43.042554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.042442    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lczw6\" (UniqueName: \"kubernetes.io/projected/13d7e931-85a3-4570-97d4-6828d7ac56a6-kube-api-access-lczw6\") pod \"13d7e931-85a3-4570-97d4-6828d7ac56a6\" (UID: \"13d7e931-85a3-4570-97d4-6828d7ac56a6\") "
Jul 09 16:34:43.042729 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.042704    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d7e931-85a3-4570-97d4-6828d7ac56a6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13d7e931-85a3-4570-97d4-6828d7ac56a6" (UID: "13d7e931-85a3-4570-97d4-6828d7ac56a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:34:43.044654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.044628    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13d7e931-85a3-4570-97d4-6828d7ac56a6" (UID: "13d7e931-85a3-4570-97d4-6828d7ac56a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:34:43.044654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.044630    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d7e931-85a3-4570-97d4-6828d7ac56a6-kube-api-access-lczw6" (OuterVolumeSpecName: "kube-api-access-lczw6") pod "13d7e931-85a3-4570-97d4-6828d7ac56a6" (UID: "13d7e931-85a3-4570-97d4-6828d7ac56a6"). InnerVolumeSpecName "kube-api-access-lczw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:43.144034 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.143995    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13d7e931-85a3-4570-97d4-6828d7ac56a6-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:43.144034 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.144025    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczw6\" (UniqueName: \"kubernetes.io/projected/13d7e931-85a3-4570-97d4-6828d7ac56a6-kube-api-access-lczw6\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:43.144034 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.144036    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13d7e931-85a3-4570-97d4-6828d7ac56a6-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:43.914928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.914871    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"
Jul 09 16:34:43.915362 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.914873    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz" event={"ID":"13d7e931-85a3-4570-97d4-6828d7ac56a6","Type":"ContainerDied","Data":"a8c4428356dc41985c832036e6cd9f997b02b8720a781ecbe23dcd0e8aca4a95"}
Jul 09 16:34:43.915362 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.915032    2558 scope.go:117] "RemoveContainer" containerID="342832a30aaa5840806a55ac132b960c10950e04f6a9cf839e80b25978caf1eb"
Jul 09 16:34:43.932804 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.932770    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"]
Jul 09 16:34:43.936006 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:43.935974    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-5a1c9889-74c4fb675f-bj8lz"]
Jul 09 16:34:45.177640 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:45.177608    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d7e931-85a3-4570-97d4-6828d7ac56a6" path="/var/lib/kubelet/pods/13d7e931-85a3-4570-97d4-6828d7ac56a6/volumes"
Jul 09 16:34:47.780917 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.780814    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:47.892100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892061    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-token\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892100    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6150320-0ea0-432e-990a-a3d62791b8d0-istiod-ca-cert\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892343 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892126    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-envoy\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892343 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892168    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-credential-socket\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892343 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892197    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-socket\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892343 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892261    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-certs\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892343 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892278    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfznw\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-kube-api-access-jfznw\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892343 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-podinfo\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892348    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-data\") pod \"f6150320-0ea0-432e-990a-a3d62791b8d0\" (UID: \"f6150320-0ea0-432e-990a-a3d62791b8d0\") "
Jul 09 16:34:47.892654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892485    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6150320-0ea0-432e-990a-a3d62791b8d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:34:47.892654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:47.892654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892627    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6150320-0ea0-432e-990a-a3d62791b8d0-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.892654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892619    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:47.892947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892658    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:47.892947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.892849    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:47.894550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.894524    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:34:47.894550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.894534    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-kube-api-access-jfznw" (OuterVolumeSpecName: "kube-api-access-jfznw") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "kube-api-access-jfznw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:47.894733 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.894577    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:34:47.894787 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.894767    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f6150320-0ea0-432e-990a-a3d62791b8d0" (UID: "f6150320-0ea0-432e-990a-a3d62791b8d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:34:47.939182 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.939146    2558 generic.go:356] "Generic (PLEG): container finished" podID="f6150320-0ea0-432e-990a-a3d62791b8d0" containerID="ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068" exitCode=0
Jul 09 16:34:47.939353 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.939197    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"
Jul 09 16:34:47.939353 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.939196    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb" event={"ID":"f6150320-0ea0-432e-990a-a3d62791b8d0","Type":"ContainerDied","Data":"ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068"}
Jul 09 16:34:47.939353 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.939329    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb" event={"ID":"f6150320-0ea0-432e-990a-a3d62791b8d0","Type":"ContainerDied","Data":"43d8ea878f28a9341f8da65a72c9d6e01632b02f4c898f75381054e10d4a82e1"}
Jul 09 16:34:47.939452 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.939360    2558 scope.go:117] "RemoveContainer" containerID="ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068"
Jul 09 16:34:47.949199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.949177    2558 scope.go:117] "RemoveContainer" containerID="ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068"
Jul 09 16:34:47.949507 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:34:47.949486    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068\": container with ID starting with ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068 not found: ID does not exist" containerID="ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068"
Jul 09 16:34:47.949559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.949516    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068"} err="failed to get container status \"ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068\": rpc error: code = NotFound desc = could not find container \"ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068\": container with ID starting with ee8f4f182be56d0cd6333bbf9c38bcae9127aee846eb5e263a4d29bc6754b068 not found: ID does not exist"
Jul 09 16:34:47.960187 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.960157    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"]
Jul 09 16:34:47.965536 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.965507    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-5a1c9889-openshift-default-77d9d7c4f5-5p9vb"]
Jul 09 16:34:47.993635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993604    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993633    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993645    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfznw\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-kube-api-access-jfznw\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993654    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993662    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993674    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993684    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:47.993869 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:47.993694    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6150320-0ea0-432e-990a-a3d62791b8d0-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:34:49.181373 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:34:49.181334    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6150320-0ea0-432e-990a-a3d62791b8d0" path="/var/lib/kubelet/pods/f6150320-0ea0-432e-990a-a3d62791b8d0/volumes"
Jul 09 16:36:22.698701 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.698664    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"]
Jul 09 16:36:22.699199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.699067    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6150320-0ea0-432e-990a-a3d62791b8d0" containerName="istio-proxy"
Jul 09 16:36:22.699199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.699081    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6150320-0ea0-432e-990a-a3d62791b8d0" containerName="istio-proxy"
Jul 09 16:36:22.699199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.699098    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d7e931-85a3-4570-97d4-6828d7ac56a6" containerName="maas-api"
Jul 09 16:36:22.699199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.699104    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d7e931-85a3-4570-97d4-6828d7ac56a6" containerName="maas-api"
Jul 09 16:36:22.699199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.699160    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6150320-0ea0-432e-990a-a3d62791b8d0" containerName="istio-proxy"
Jul 09 16:36:22.699199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.699170    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d7e931-85a3-4570-97d4-6828d7ac56a6" containerName="maas-api"
Jul 09 16:36:22.702122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.702103    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.705101 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.705082    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f14f1e2b-openshift-default-dockercfg-w89db\""
Jul 09 16:36:22.723310 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.723279    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"]
Jul 09 16:36:22.830376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-credential-socket\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830376 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830379    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-socket\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830410    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istiod-ca-cert\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830474    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p7vd\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-kube-api-access-8p7vd\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830519    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-certs\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830544    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-data\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830567    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-podinfo\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830582    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-envoy\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.830794 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.830597    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-token\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istiod-ca-cert\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932156    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p7vd\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-kube-api-access-8p7vd\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932233    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-certs\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932257    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-data\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932392    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-podinfo\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932440    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-envoy\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-token\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932534    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-credential-socket\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932561    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-socket\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932642    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-data\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932857 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-certs\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932965 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932944    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istiod-ca-cert\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.932965 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932954    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-socket\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.933037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.932969    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-credential-socket\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.934904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.934863    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-envoy\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.935239 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.935217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-podinfo\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.951099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.951021    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p7vd\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-kube-api-access-8p7vd\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:22.960460 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:22.960428    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-token\") pod \"e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") " pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:23.013926 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.013871    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:23.164972 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.164941    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"]
Jul 09 16:36:23.167396 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:36:23.167366    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39231fb2_2cfe_4ae4_85e0_6ac4f5027182.slice/crio-71288186971d6405e252b38ee016e1e0b846fa57e2da67695ba15ca78bbad173 WatchSource:0}: Error finding container 71288186971d6405e252b38ee016e1e0b846fa57e2da67695ba15ca78bbad173: Status 404 returned error can't find the container with id 71288186971d6405e252b38ee016e1e0b846fa57e2da67695ba15ca78bbad173
Jul 09 16:36:23.169790 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.169760    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:36:23.169857 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.169830    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:36:23.169913 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.169860    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:36:23.438902 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.438839    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" event={"ID":"39231fb2-2cfe-4ae4-85e0-6ac4f5027182","Type":"ContainerStarted","Data":"6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"}
Jul 09 16:36:23.439099 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.438924    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" event={"ID":"39231fb2-2cfe-4ae4-85e0-6ac4f5027182","Type":"ContainerStarted","Data":"71288186971d6405e252b38ee016e1e0b846fa57e2da67695ba15ca78bbad173"}
Jul 09 16:36:23.460333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:23.460210    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" podStartSLOduration=1.460194406 podStartE2EDuration="1.460194406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:36:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:36:23.459533907 +0000 UTC m=+1928.841350695" watchObservedRunningTime="2026-07-09 16:36:23.460194406 +0000 UTC m=+1928.842011194"
Jul 09 16:36:24.014485 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:24.014445    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:24.020361 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:24.020329    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:24.444418 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:24.444387    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:24.445468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:24.445449    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:25.475588 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.475559    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"]
Jul 09 16:36:25.479039 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.479017    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.481944 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.481922    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:36:25.481944 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.481939    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:36:25.483224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.483205    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:36:25.483331 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.483232    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f14f1e2b-serving-cert\""
Jul 09 16:36:25.490282 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.490259    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"]
Jul 09 16:36:25.557548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.557511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cjtj\" (UniqueName: \"kubernetes.io/projected/d585cca8-5466-479b-8b40-3f53523e1d4c-kube-api-access-9cjtj\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.557739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.557689    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d585cca8-5466-479b-8b40-3f53523e1d4c-maas-api-tls\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.557739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.557715    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d585cca8-5466-479b-8b40-3f53523e1d4c-openshift-service-ca\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.659072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.659036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d585cca8-5466-479b-8b40-3f53523e1d4c-maas-api-tls\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.659274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.659086    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d585cca8-5466-479b-8b40-3f53523e1d4c-openshift-service-ca\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.659274 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.659169    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cjtj\" (UniqueName: \"kubernetes.io/projected/d585cca8-5466-479b-8b40-3f53523e1d4c-kube-api-access-9cjtj\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.659891 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.659855    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d585cca8-5466-479b-8b40-3f53523e1d4c-openshift-service-ca\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.661853 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.661828    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d585cca8-5466-479b-8b40-3f53523e1d4c-maas-api-tls\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.669416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.669376    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cjtj\" (UniqueName: \"kubernetes.io/projected/d585cca8-5466-479b-8b40-3f53523e1d4c-kube-api-access-9cjtj\") pod \"maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.791212 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.791118    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:25.941069 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.941037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"]
Jul 09 16:36:25.943022 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:36:25.942992    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd585cca8_5466_479b_8b40_3f53523e1d4c.slice/crio-ab2301a47c834f5090be8a320c49a758c226cb22910ed974b5d089128dbc5ec3 WatchSource:0}: Error finding container ab2301a47c834f5090be8a320c49a758c226cb22910ed974b5d089128dbc5ec3: Status 404 returned error can't find the container with id ab2301a47c834f5090be8a320c49a758c226cb22910ed974b5d089128dbc5ec3
Jul 09 16:36:25.945240 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:25.945221    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:36:26.457508 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:26.457463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf" event={"ID":"d585cca8-5466-479b-8b40-3f53523e1d4c","Type":"ContainerStarted","Data":"2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4"}
Jul 09 16:36:26.457725 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:26.457541    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf" event={"ID":"d585cca8-5466-479b-8b40-3f53523e1d4c","Type":"ContainerStarted","Data":"ab2301a47c834f5090be8a320c49a758c226cb22910ed974b5d089128dbc5ec3"}
Jul 09 16:36:26.496321 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:26.496265    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf" podStartSLOduration=1.074030601 podStartE2EDuration="1.496239334s" totalImagesPullingTime="422.208733ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:36:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:36:26.495397701 +0000 UTC m=+1931.877214491" watchObservedRunningTime="2026-07-09 16:36:26.496239334 +0000 UTC m=+1931.878056123"
Jul 09 16:36:27.463102 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:27.463064    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:33.475104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:33.475074    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:37.430195 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.430155    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"]
Jul 09 16:36:37.431112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.430398    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf" podUID="d585cca8-5466-479b-8b40-3f53523e1d4c" containerName="maas-api" containerID="cri-o://2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4" gracePeriod=30
Jul 09 16:36:37.708158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.708120    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:37.775827 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.775786    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cjtj\" (UniqueName: \"kubernetes.io/projected/d585cca8-5466-479b-8b40-3f53523e1d4c-kube-api-access-9cjtj\") pod \"d585cca8-5466-479b-8b40-3f53523e1d4c\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") "
Jul 09 16:36:37.776049 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.775872    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d585cca8-5466-479b-8b40-3f53523e1d4c-openshift-service-ca\") pod \"d585cca8-5466-479b-8b40-3f53523e1d4c\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") "
Jul 09 16:36:37.776049 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.775932    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d585cca8-5466-479b-8b40-3f53523e1d4c-maas-api-tls\") pod \"d585cca8-5466-479b-8b40-3f53523e1d4c\" (UID: \"d585cca8-5466-479b-8b40-3f53523e1d4c\") "
Jul 09 16:36:37.776366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.776329    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d585cca8-5466-479b-8b40-3f53523e1d4c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d585cca8-5466-479b-8b40-3f53523e1d4c" (UID: "d585cca8-5466-479b-8b40-3f53523e1d4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:36:37.778229 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.778195    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d585cca8-5466-479b-8b40-3f53523e1d4c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d585cca8-5466-479b-8b40-3f53523e1d4c" (UID: "d585cca8-5466-479b-8b40-3f53523e1d4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:36:37.778336 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.778279    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d585cca8-5466-479b-8b40-3f53523e1d4c-kube-api-access-9cjtj" (OuterVolumeSpecName: "kube-api-access-9cjtj") pod "d585cca8-5466-479b-8b40-3f53523e1d4c" (UID: "d585cca8-5466-479b-8b40-3f53523e1d4c"). InnerVolumeSpecName "kube-api-access-9cjtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:36:37.877248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.877210    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cjtj\" (UniqueName: \"kubernetes.io/projected/d585cca8-5466-479b-8b40-3f53523e1d4c-kube-api-access-9cjtj\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:37.877248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.877241    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d585cca8-5466-479b-8b40-3f53523e1d4c-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:37.877248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:37.877251    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d585cca8-5466-479b-8b40-3f53523e1d4c-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:38.541549 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.541509    2558 generic.go:356] "Generic (PLEG): container finished" podID="d585cca8-5466-479b-8b40-3f53523e1d4c" containerID="2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4" exitCode=0
Jul 09 16:36:38.542000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.541566    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"
Jul 09 16:36:38.542000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.541599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf" event={"ID":"d585cca8-5466-479b-8b40-3f53523e1d4c","Type":"ContainerDied","Data":"2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4"}
Jul 09 16:36:38.542000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.541667    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf" event={"ID":"d585cca8-5466-479b-8b40-3f53523e1d4c","Type":"ContainerDied","Data":"ab2301a47c834f5090be8a320c49a758c226cb22910ed974b5d089128dbc5ec3"}
Jul 09 16:36:38.542000 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.541702    2558 scope.go:117] "RemoveContainer" containerID="2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4"
Jul 09 16:36:38.553178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.553079    2558 scope.go:117] "RemoveContainer" containerID="2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4"
Jul 09 16:36:38.553490 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:36:38.553458    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4\": container with ID starting with 2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4 not found: ID does not exist" containerID="2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4"
Jul 09 16:36:38.553621 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.553495    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4"} err="failed to get container status \"2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4\": rpc error: code = NotFound desc = could not find container \"2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4\": container with ID starting with 2fb6072e12d87104a664400b1a29dc4ffcbfb4a9f7fc6c4d6775e91802bf6fa4 not found: ID does not exist"
Jul 09 16:36:38.564504 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.564468    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"]
Jul 09 16:36:38.568691 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:38.568661    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f14f1e2b-7cc7d6f5f8-wwxnf"]
Jul 09 16:36:39.178377 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:39.178332    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d585cca8-5466-479b-8b40-3f53523e1d4c" path="/var/lib/kubelet/pods/d585cca8-5466-479b-8b40-3f53523e1d4c/volumes"
Jul 09 16:36:39.692560 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:39.692524    2558 patch_prober.go:28] interesting pod/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:57748->10.134.0.54:15021: read: connection reset by peer" start-of-body=
Jul 09 16:36:39.693016 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:39.692607    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:57748->10.134.0.54:15021: read: connection reset by peer"
Jul 09 16:36:40.554805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:40.554772    2558 generic.go:356] "Generic (PLEG): container finished" podID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerID="6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f" exitCode=0
Jul 09 16:36:40.555023 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:40.554853    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" event={"ID":"39231fb2-2cfe-4ae4-85e0-6ac4f5027182","Type":"ContainerDied","Data":"6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"}
Jul 09 16:36:40.555325 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:40.555310    2558 scope.go:117] "RemoveContainer" containerID="6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"
Jul 09 16:36:40.556331 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:40.556303    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:36:40.556413 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:40.556384    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:36:40.556450 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:40.556426    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:36:41.562409 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:41.562373    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" event={"ID":"39231fb2-2cfe-4ae4-85e0-6ac4f5027182","Type":"ContainerStarted","Data":"a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3"}
Jul 09 16:36:42.014941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:42.014901    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:42.015247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:42.014960    2558 patch_prober.go:28] interesting pod/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 09 16:36:42.015428 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:42.015398    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 09 16:36:43.015307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:43.015262    2558 patch_prober.go:28] interesting pod/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body=
Jul 09 16:36:43.015932 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:43.015864    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused"
Jul 09 16:36:43.469646 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:43.469608    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"]
Jul 09 16:36:43.575270 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:43.575225    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy" containerID="cri-o://a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3" gracePeriod=30
Jul 09 16:36:50.848723 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.848693    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:50.902692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902596    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-certs\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.902692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902672    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-podinfo\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.902938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902714    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-socket\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.902938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902748    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-data\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.902938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902894    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istiod-ca-cert\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.903104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902962    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:36:50.903104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.902983    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-credential-socket\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.903104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8p7vd\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-kube-api-access-8p7vd\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.903104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903057    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-token\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.903104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903087    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-envoy\") pod \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\" (UID: \"39231fb2-2cfe-4ae4-85e0-6ac4f5027182\") "
Jul 09 16:36:50.903374 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903079    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:36:50.903374 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903176    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:36:50.903481 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903382    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:50.903481 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903402    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:50.903481 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903416    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:50.903635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903606    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-data" (OuterVolumeSpecName: "istio-data") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:36:50.903635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.903617    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:36:50.905530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.905505    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-token" (OuterVolumeSpecName: "istio-token") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:36:50.905831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.905802    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:36:50.905831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.905807    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-kube-api-access-8p7vd" (OuterVolumeSpecName: "kube-api-access-8p7vd") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "kube-api-access-8p7vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:36:50.906053 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:50.905810    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39231fb2-2cfe-4ae4-85e0-6ac4f5027182" (UID: "39231fb2-2cfe-4ae4-85e0-6ac4f5027182"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:36:51.004761 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.004719    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:51.004761 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.004757    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:51.004761 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.004767    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p7vd\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-kube-api-access-8p7vd\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:51.005048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.004776    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:51.005048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.004785    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:51.005048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.004792    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39231fb2-2cfe-4ae4-85e0-6ac4f5027182-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:36:51.620472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.620438    2558 generic.go:356] "Generic (PLEG): container finished" podID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerID="a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3" exitCode=0
Jul 09 16:36:51.620698 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.620500    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"
Jul 09 16:36:51.620698 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.620527    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" event={"ID":"39231fb2-2cfe-4ae4-85e0-6ac4f5027182","Type":"ContainerDied","Data":"a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3"}
Jul 09 16:36:51.620698 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.620590    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk" event={"ID":"39231fb2-2cfe-4ae4-85e0-6ac4f5027182","Type":"ContainerDied","Data":"71288186971d6405e252b38ee016e1e0b846fa57e2da67695ba15ca78bbad173"}
Jul 09 16:36:51.620698 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.620614    2558 scope.go:117] "RemoveContainer" containerID="a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3"
Jul 09 16:36:51.629770 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.629754    2558 scope.go:117] "RemoveContainer" containerID="6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"
Jul 09 16:36:51.638168 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.638117    2558 scope.go:117] "RemoveContainer" containerID="a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3"
Jul 09 16:36:51.638630 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:36:51.638602    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3\": container with ID starting with a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3 not found: ID does not exist" containerID="a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3"
Jul 09 16:36:51.638739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.638640    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3"} err="failed to get container status \"a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3\": rpc error: code = NotFound desc = could not find container \"a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3\": container with ID starting with a4c22849894d250cff0dfe58df2981f6788908341e12fca5a5b2b5beb83390f3 not found: ID does not exist"
Jul 09 16:36:51.638739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.638666    2558 scope.go:117] "RemoveContainer" containerID="6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"
Jul 09 16:36:51.638956 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:36:51.638936    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f\": container with ID starting with 6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f not found: ID does not exist" containerID="6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"
Jul 09 16:36:51.639024 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.638977    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f"} err="failed to get container status \"6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f\": rpc error: code = NotFound desc = could not find container \"6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f\": container with ID starting with 6cb23abcf4c1b78daf180662567fd80af6ff82f26fa9fc8d5adfe01ecddc7c5f not found: ID does not exist"
Jul 09 16:36:51.640911 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.640863    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"]
Jul 09 16:36:51.646370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:51.646348    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f14f1e2b-openshift-default-64587964cf-ml5wk"]
Jul 09 16:36:53.178039 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:36:53.177989    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" path="/var/lib/kubelet/pods/39231fb2-2cfe-4ae4-85e0-6ac4f5027182/volumes"
Jul 09 16:37:02.648721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.648682    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"]
Jul 09 16:37:02.649332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649303    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d585cca8-5466-479b-8b40-3f53523e1d4c" containerName="maas-api"
Jul 09 16:37:02.649332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649328    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d585cca8-5466-479b-8b40-3f53523e1d4c" containerName="maas-api"
Jul 09 16:37:02.649468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649346    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy"
Jul 09 16:37:02.649468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649354    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy"
Jul 09 16:37:02.649468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649380    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy"
Jul 09 16:37:02.649468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649388    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy"
Jul 09 16:37:02.649657 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649475    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy"
Jul 09 16:37:02.649657 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649491    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="39231fb2-2cfe-4ae4-85e0-6ac4f5027182" containerName="istio-proxy"
Jul 09 16:37:02.649657 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.649502    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d585cca8-5466-479b-8b40-3f53523e1d4c" containerName="maas-api"
Jul 09 16:37:02.654311 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.654287    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.657636 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.657615    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-85fe79d8-openshift-default-dockercfg-zgnr2\""
Jul 09 16:37:02.663344 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.663320    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"]
Jul 09 16:37:02.714054 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714013    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istiod-ca-cert\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714062    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxgdk\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-kube-api-access-jxgdk\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714133    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-certs\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714162    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-data\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714277    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-envoy\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714329    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-credential-socket\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714361    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-podinfo\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714401    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-socket\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.714571 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.714425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-token\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.815935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.815870    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-socket\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.815943    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-token\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.815989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istiod-ca-cert\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxgdk\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-kube-api-access-jxgdk\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816104    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-certs\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816140    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-data\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816432 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816167    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-envoy\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816432 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816203    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-credential-socket\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816432 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816229    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-podinfo\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816432 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816333    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-socket\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816630    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-credential-socket\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-certs\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816684    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-data\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.816818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.816781    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istiod-ca-cert\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.818786 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.818765    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-envoy\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.819297 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.819274    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-podinfo\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.832523 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.832488    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxgdk\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-kube-api-access-jxgdk\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.833100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.833069    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-token\") pod \"e2e-mt-85fe79d8-openshift-default-957b4475-7kthv\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") " pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:02.966314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:02.966204    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:03.159731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.159532    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"]
Jul 09 16:37:03.162041 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:37:03.162012    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc7da2c_769c_4483_a124_a5c1d2dfd338.slice/crio-944a3d5b3c24f3888d06076543b24e61d165963c5334bde9e8dc84b20debe552 WatchSource:0}: Error finding container 944a3d5b3c24f3888d06076543b24e61d165963c5334bde9e8dc84b20debe552: Status 404 returned error can't find the container with id 944a3d5b3c24f3888d06076543b24e61d165963c5334bde9e8dc84b20debe552
Jul 09 16:37:03.164312 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.164278    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:37:03.164420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.164349    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:37:03.164420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.164402    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:37:03.691673 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.691632    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" event={"ID":"7cc7da2c-769c-4483-a124-a5c1d2dfd338","Type":"ContainerStarted","Data":"8960e837dda5f2a90716e954710d277dd694deef820421997dee09d3ec006cb8"}
Jul 09 16:37:03.692193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.691699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" event={"ID":"7cc7da2c-769c-4483-a124-a5c1d2dfd338","Type":"ContainerStarted","Data":"944a3d5b3c24f3888d06076543b24e61d165963c5334bde9e8dc84b20debe552"}
Jul 09 16:37:03.713564 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.713508    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" podStartSLOduration=1.713484331 podStartE2EDuration="1.713484331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:37:03.711332558 +0000 UTC m=+1969.093149345" watchObservedRunningTime="2026-07-09 16:37:03.713484331 +0000 UTC m=+1969.095301119"
Jul 09 16:37:03.966742 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:03.966653    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:04.972062 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:04.971970    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:05.703199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:05.703163    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:05.704218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:05.704192    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:06.338550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.338516    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"]
Jul 09 16:37:06.342011 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.341988    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.345117 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.345092    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-85fe79d8-serving-cert\""
Jul 09 16:37:06.345234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.345176    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:37:06.346624 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.346604    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:37:06.346761 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.346635    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:37:06.353393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.353367    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"]
Jul 09 16:37:06.451512 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.451471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.451694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.451526    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr25m\" (UniqueName: \"kubernetes.io/projected/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-kube-api-access-tr25m\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.451694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.451548    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-openshift-service-ca\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.552529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.552480    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr25m\" (UniqueName: \"kubernetes.io/projected/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-kube-api-access-tr25m\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.552529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.552534    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-openshift-service-ca\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.552817 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.552657    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.552817 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:37:06.552783    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-serving-cert: secret "maas-api-e2e-mt-85fe79d8-serving-cert" not found
Jul 09 16:37:06.552968 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:37:06.552856    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls podName:fedb83e6-b529-41e3-85a4-e54b7dea3ccc nodeName:}" failed. No retries permitted until 2026-07-09 16:37:07.052832082 +0000 UTC m=+1972.434648857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls") pod "maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" (UID: "fedb83e6-b529-41e3-85a4-e54b7dea3ccc") : secret "maas-api-e2e-mt-85fe79d8-serving-cert" not found
Jul 09 16:37:06.553232 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.553209    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-openshift-service-ca\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:06.561488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:06.561465    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr25m\" (UniqueName: \"kubernetes.io/projected/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-kube-api-access-tr25m\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:07.058521 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:07.058483    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:07.061209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:07.061177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls\") pod \"maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:07.254629 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:07.254588    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:07.402012 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:07.401982    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"]
Jul 09 16:37:07.403741 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:37:07.403709    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfedb83e6_b529_41e3_85a4_e54b7dea3ccc.slice/crio-8019bce7258b5e346b04c987aba038437822ad71f7f4a7b6d0785d7c0a576111 WatchSource:0}: Error finding container 8019bce7258b5e346b04c987aba038437822ad71f7f4a7b6d0785d7c0a576111: Status 404 returned error can't find the container with id 8019bce7258b5e346b04c987aba038437822ad71f7f4a7b6d0785d7c0a576111
Jul 09 16:37:07.716144 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:07.716043    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" event={"ID":"fedb83e6-b529-41e3-85a4-e54b7dea3ccc","Type":"ContainerStarted","Data":"8019bce7258b5e346b04c987aba038437822ad71f7f4a7b6d0785d7c0a576111"}
Jul 09 16:37:08.722530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:08.722493    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" event={"ID":"fedb83e6-b529-41e3-85a4-e54b7dea3ccc","Type":"ContainerStarted","Data":"2c478bdeae122eaae14d817d60d5a6f9c6081be8c9d5535af1a444e06066e758"}
Jul 09 16:37:08.722990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:08.722678    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:08.740862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:08.740815    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" podStartSLOduration=2.313378903 podStartE2EDuration="2.740801108s" totalImagesPullingTime="427.422205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:37:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:37:08.738199396 +0000 UTC m=+1974.120016184" watchObservedRunningTime="2026-07-09 16:37:08.740801108 +0000 UTC m=+1974.122617896"
Jul 09 16:37:14.733558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:14.733521    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:17.777184 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:17.777085    2558 generic.go:356] "Generic (PLEG): container finished" podID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerID="8960e837dda5f2a90716e954710d277dd694deef820421997dee09d3ec006cb8" exitCode=0
Jul 09 16:37:17.777184 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:17.777155    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" event={"ID":"7cc7da2c-769c-4483-a124-a5c1d2dfd338","Type":"ContainerDied","Data":"8960e837dda5f2a90716e954710d277dd694deef820421997dee09d3ec006cb8"}
Jul 09 16:37:17.777609 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:17.777572    2558 scope.go:117] "RemoveContainer" containerID="8960e837dda5f2a90716e954710d277dd694deef820421997dee09d3ec006cb8"
Jul 09 16:37:17.778614 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:17.778583    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:37:17.778710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:17.778653    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:37:17.778710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:17.778684    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:37:18.166483 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.166450    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"]
Jul 09 16:37:18.169759 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.169734    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.172763 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.172738    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-85fe79d8\"/\"kube-root-ca.crt\""
Jul 09 16:37:18.172928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.172780    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-85fe79d8\"/\"default-dockercfg-xcjs5\""
Jul 09 16:37:18.172928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.172794    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-85fe79d8\"/\"e2e-lifecycle-model-85fe79d8-kserve-self-signed-certs\""
Jul 09 16:37:18.172928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.172738    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-85fe79d8\"/\"openshift-service-ca.crt\""
Jul 09 16:37:18.179218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.179178    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"]
Jul 09 16:37:18.272425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.272389    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-tmp-dir\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.272425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.272429    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6827\" (UniqueName: \"kubernetes.io/projected/f91a4eba-d555-466f-afe1-2764192f148f-kube-api-access-m6827\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.272700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.272518    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-home\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.272700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.272589    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-dshm\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.272700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.272612    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91a4eba-d555-466f-afe1-2764192f148f-tls-certs\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.272700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.272642    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-model-cache\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-tmp-dir\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373411    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6827\" (UniqueName: \"kubernetes.io/projected/f91a4eba-d555-466f-afe1-2764192f148f-kube-api-access-m6827\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373481    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-home\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373517    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-dshm\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373539    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91a4eba-d555-466f-afe1-2764192f148f-tls-certs\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373592 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373564    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-model-cache\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.373925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373861    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-tmp-dir\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.374049 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.373988    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-home\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.374122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.374066    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-model-cache\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.376078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.376057    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-dshm\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.376196 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.376177    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91a4eba-d555-466f-afe1-2764192f148f-tls-certs\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.384713 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.384690    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6827\" (UniqueName: \"kubernetes.io/projected/f91a4eba-d555-466f-afe1-2764192f148f-kube-api-access-m6827\") pod \"e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") " pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.481260 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.481160    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:18.666339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.666298    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"]
Jul 09 16:37:18.667758 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:37:18.667731    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf91a4eba_d555_466f_afe1_2764192f148f.slice/crio-5402275e6a49c295fa6ab30cb89493e9ee77f067ddae0b36d59a15c4dbebe6b1 WatchSource:0}: Error finding container 5402275e6a49c295fa6ab30cb89493e9ee77f067ddae0b36d59a15c4dbebe6b1: Status 404 returned error can't find the container with id 5402275e6a49c295fa6ab30cb89493e9ee77f067ddae0b36d59a15c4dbebe6b1
Jul 09 16:37:18.784348 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.784313    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" event={"ID":"7cc7da2c-769c-4483-a124-a5c1d2dfd338","Type":"ContainerStarted","Data":"b93d84eb3ef0115fa87886ee125baaa6a94a1cdd3dfc8cbb5c3648a71240bea4"}
Jul 09 16:37:18.786186 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.786158    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb" event={"ID":"f91a4eba-d555-466f-afe1-2764192f148f","Type":"ContainerStarted","Data":"c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9"}
Jul 09 16:37:18.786321 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.786201    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb" event={"ID":"f91a4eba-d555-466f-afe1-2764192f148f","Type":"ContainerStarted","Data":"5402275e6a49c295fa6ab30cb89493e9ee77f067ddae0b36d59a15c4dbebe6b1"}
Jul 09 16:37:18.825914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.823950    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb" podStartSLOduration=0.823906646 podStartE2EDuration="823.906646ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:37:18.822342409 +0000 UTC m=+1984.204159198" watchObservedRunningTime="2026-07-09 16:37:18.823906646 +0000 UTC m=+1984.205723434"
Jul 09 16:37:18.967372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.967336    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:18.968696 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.968665    2558 patch_prober.go:28] interesting pod/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 09 16:37:18.968927 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:18.968717    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 09 16:37:19.967116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:19.967072    2558 patch_prober.go:28] interesting pod/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 09 16:37:19.967516 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:19.967133    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 09 16:37:20.703105 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:20.703076    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:20.967724 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:20.967627    2558 patch_prober.go:28] interesting pod/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 09 16:37:20.967724 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:20.967695    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 09 16:37:21.967165 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:21.967121    2558 patch_prober.go:28] interesting pod/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Jul 09 16:37:21.967368 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:21.967198    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Jul 09 16:37:22.970748 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:22.970709    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:22.971706 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:22.971683    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:37:28.482143 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:28.482090    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:28.482143 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:28.482144    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:28.494988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:28.494951    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:28.864397 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:28.864362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:37:42.763427 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:42.763388    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"]
Jul 09 16:37:42.764041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:42.763674    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" podUID="fedb83e6-b529-41e3-85a4-e54b7dea3ccc" containerName="maas-api" containerID="cri-o://2c478bdeae122eaae14d817d60d5a6f9c6081be8c9d5535af1a444e06066e758" gracePeriod=30
Jul 09 16:37:42.927995 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:42.927958    2558 generic.go:356] "Generic (PLEG): container finished" podID="fedb83e6-b529-41e3-85a4-e54b7dea3ccc" containerID="2c478bdeae122eaae14d817d60d5a6f9c6081be8c9d5535af1a444e06066e758" exitCode=0
Jul 09 16:37:42.928181 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:42.928024    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" event={"ID":"fedb83e6-b529-41e3-85a4-e54b7dea3ccc","Type":"ContainerDied","Data":"2c478bdeae122eaae14d817d60d5a6f9c6081be8c9d5535af1a444e06066e758"}
Jul 09 16:37:43.027121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.027050    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:43.119019 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.118965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls\") pod \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") "
Jul 09 16:37:43.119224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.119080    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr25m\" (UniqueName: \"kubernetes.io/projected/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-kube-api-access-tr25m\") pod \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") "
Jul 09 16:37:43.119224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.119146    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-openshift-service-ca\") pod \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\" (UID: \"fedb83e6-b529-41e3-85a4-e54b7dea3ccc\") "
Jul 09 16:37:43.119450 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.119426    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fedb83e6-b529-41e3-85a4-e54b7dea3ccc" (UID: "fedb83e6-b529-41e3-85a4-e54b7dea3ccc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:37:43.121252 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.121220    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fedb83e6-b529-41e3-85a4-e54b7dea3ccc" (UID: "fedb83e6-b529-41e3-85a4-e54b7dea3ccc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:37:43.121350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.121280    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-kube-api-access-tr25m" (OuterVolumeSpecName: "kube-api-access-tr25m") pod "fedb83e6-b529-41e3-85a4-e54b7dea3ccc" (UID: "fedb83e6-b529-41e3-85a4-e54b7dea3ccc"). InnerVolumeSpecName "kube-api-access-tr25m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:37:43.220510 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.220470    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:37:43.220510 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.220504    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:37:43.220510 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.220513    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr25m\" (UniqueName: \"kubernetes.io/projected/fedb83e6-b529-41e3-85a4-e54b7dea3ccc-kube-api-access-tr25m\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:37:43.934807 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.934777    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"
Jul 09 16:37:43.935350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.934772    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w" event={"ID":"fedb83e6-b529-41e3-85a4-e54b7dea3ccc","Type":"ContainerDied","Data":"8019bce7258b5e346b04c987aba038437822ad71f7f4a7b6d0785d7c0a576111"}
Jul 09 16:37:43.935350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.934936    2558 scope.go:117] "RemoveContainer" containerID="2c478bdeae122eaae14d817d60d5a6f9c6081be8c9d5535af1a444e06066e758"
Jul 09 16:37:43.952929 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.952900    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"]
Jul 09 16:37:43.956424 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:43.956397    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-85fe79d8-6787d88d99-qvl5w"]
Jul 09 16:37:45.178294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:45.178260    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fedb83e6-b529-41e3-85a4-e54b7dea3ccc" path="/var/lib/kubelet/pods/fedb83e6-b529-41e3-85a4-e54b7dea3ccc/volumes"
Jul 09 16:37:53.481731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:37:53.481690    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"]
Jul 09 16:38:08.504033 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.503979    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb" podUID="f91a4eba-d555-466f-afe1-2764192f148f" containerName="main" containerID="cri-o://c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9" gracePeriod=45
Jul 09 16:38:08.753651 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.753619    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:38:08.866636 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866603    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6827\" (UniqueName: \"kubernetes.io/projected/f91a4eba-d555-466f-afe1-2764192f148f-kube-api-access-m6827\") pod \"f91a4eba-d555-466f-afe1-2764192f148f\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") "
Jul 09 16:38:08.866636 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866639    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91a4eba-d555-466f-afe1-2764192f148f-tls-certs\") pod \"f91a4eba-d555-466f-afe1-2764192f148f\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") "
Jul 09 16:38:08.866949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866705    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-model-cache\") pod \"f91a4eba-d555-466f-afe1-2764192f148f\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") "
Jul 09 16:38:08.866949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866788    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-tmp-dir\") pod \"f91a4eba-d555-466f-afe1-2764192f148f\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") "
Jul 09 16:38:08.866949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866850    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-home\") pod \"f91a4eba-d555-466f-afe1-2764192f148f\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") "
Jul 09 16:38:08.866949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866870    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-dshm\") pod \"f91a4eba-d555-466f-afe1-2764192f148f\" (UID: \"f91a4eba-d555-466f-afe1-2764192f148f\") "
Jul 09 16:38:08.866949 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.866919    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-model-cache" (OuterVolumeSpecName: "model-cache") pod "f91a4eba-d555-466f-afe1-2764192f148f" (UID: "f91a4eba-d555-466f-afe1-2764192f148f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:08.867213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.867096    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f91a4eba-d555-466f-afe1-2764192f148f" (UID: "f91a4eba-d555-466f-afe1-2764192f148f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:08.867213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.867120    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-home" (OuterVolumeSpecName: "home") pod "f91a4eba-d555-466f-afe1-2764192f148f" (UID: "f91a4eba-d555-466f-afe1-2764192f148f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:08.867213 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.867201    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:08.867357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.867219    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:08.867357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.867229    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:08.869135 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.869110    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-dshm" (OuterVolumeSpecName: "dshm") pod "f91a4eba-d555-466f-afe1-2764192f148f" (UID: "f91a4eba-d555-466f-afe1-2764192f148f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:08.869135 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.869120    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91a4eba-d555-466f-afe1-2764192f148f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f91a4eba-d555-466f-afe1-2764192f148f" (UID: "f91a4eba-d555-466f-afe1-2764192f148f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:38:08.869276 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.869139    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91a4eba-d555-466f-afe1-2764192f148f-kube-api-access-m6827" (OuterVolumeSpecName: "kube-api-access-m6827") pod "f91a4eba-d555-466f-afe1-2764192f148f" (UID: "f91a4eba-d555-466f-afe1-2764192f148f"). InnerVolumeSpecName "kube-api-access-m6827". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:38:08.968307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.968261    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f91a4eba-d555-466f-afe1-2764192f148f-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:08.968307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.968299    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6827\" (UniqueName: \"kubernetes.io/projected/f91a4eba-d555-466f-afe1-2764192f148f-kube-api-access-m6827\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:08.968307 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:08.968311    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f91a4eba-d555-466f-afe1-2764192f148f-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:09.069862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.069825    2558 generic.go:356] "Generic (PLEG): container finished" podID="f91a4eba-d555-466f-afe1-2764192f148f" containerID="c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9" exitCode=0
Jul 09 16:38:09.069862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.069892    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"
Jul 09 16:38:09.070105 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.069869    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb" event={"ID":"f91a4eba-d555-466f-afe1-2764192f148f","Type":"ContainerDied","Data":"c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9"}
Jul 09 16:38:09.070105 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.069946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb" event={"ID":"f91a4eba-d555-466f-afe1-2764192f148f","Type":"ContainerDied","Data":"5402275e6a49c295fa6ab30cb89493e9ee77f067ddae0b36d59a15c4dbebe6b1"}
Jul 09 16:38:09.070105 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.069970    2558 scope.go:117] "RemoveContainer" containerID="c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9"
Jul 09 16:38:09.079575 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.079550    2558 scope.go:117] "RemoveContainer" containerID="c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9"
Jul 09 16:38:09.079916 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:38:09.079866    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9\": container with ID starting with c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9 not found: ID does not exist" containerID="c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9"
Jul 09 16:38:09.080013 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.079917    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9"} err="failed to get container status \"c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9\": rpc error: code = NotFound desc = could not find container \"c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9\": container with ID starting with c9c0a1668794bba40d2731b9748f521c9d41658dcb80264c81504d3342f7cfa9 not found: ID does not exist"
Jul 09 16:38:09.092517 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.092485    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"]
Jul 09 16:38:09.101295 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.097140    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-85fe79d8/e2e-lifecycle-model-85fe79d8-kserve-5554f89cdc-pvkgb"]
Jul 09 16:38:09.178871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:09.178781    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f91a4eba-d555-466f-afe1-2764192f148f" path="/var/lib/kubelet/pods/f91a4eba-d555-466f-afe1-2764192f148f/volumes"
Jul 09 16:38:47.057836 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:47.057795    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"]
Jul 09 16:38:47.058347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:47.058038    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy" containerID="cri-o://b93d84eb3ef0115fa87886ee125baaa6a94a1cdd3dfc8cbb5c3648a71240bea4" gracePeriod=30
Jul 09 16:38:52.298167 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.298130    2558 generic.go:356] "Generic (PLEG): container finished" podID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerID="b93d84eb3ef0115fa87886ee125baaa6a94a1cdd3dfc8cbb5c3648a71240bea4" exitCode=0
Jul 09 16:38:52.298553 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.298198    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" event={"ID":"7cc7da2c-769c-4483-a124-a5c1d2dfd338","Type":"ContainerDied","Data":"b93d84eb3ef0115fa87886ee125baaa6a94a1cdd3dfc8cbb5c3648a71240bea4"}
Jul 09 16:38:52.298553 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.298266    2558 scope.go:117] "RemoveContainer" containerID="8960e837dda5f2a90716e954710d277dd694deef820421997dee09d3ec006cb8"
Jul 09 16:38:52.333079 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.333013    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:38:52.464906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.464857    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istiod-ca-cert\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.464906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.464908    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-podinfo\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.464906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.464931    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-data\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.464952    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-token\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.464984    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-certs\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465011    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-envoy\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465045    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-socket\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxgdk\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-kube-api-access-jxgdk\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465210    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-credential-socket\") pod \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\" (UID: \"7cc7da2c-769c-4483-a124-a5c1d2dfd338\") "
Jul 09 16:38:52.465591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465322    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:38:52.465591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465492    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:52.465591 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465513    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.465766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465626    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:52.465822 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465794    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:52.465932 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.465900    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:52.467598 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.467575    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:38:52.467820 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.467795    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:38:52.468118 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.468094    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:38:52.468249 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.468230    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-kube-api-access-jxgdk" (OuterVolumeSpecName: "kube-api-access-jxgdk") pod "7cc7da2c-769c-4483-a124-a5c1d2dfd338" (UID: "7cc7da2c-769c-4483-a124-a5c1d2dfd338"). InnerVolumeSpecName "kube-api-access-jxgdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:38:52.566914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566857    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.566914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566907    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.566914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566916    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.566914 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566925    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.567193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566933    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.567193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566941    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.567193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566948    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxgdk\" (UniqueName: \"kubernetes.io/projected/7cc7da2c-769c-4483-a124-a5c1d2dfd338-kube-api-access-jxgdk\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:52.567193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:52.566959    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cc7da2c-769c-4483-a124-a5c1d2dfd338-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:38:53.307904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:53.307615    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv" event={"ID":"7cc7da2c-769c-4483-a124-a5c1d2dfd338","Type":"ContainerDied","Data":"944a3d5b3c24f3888d06076543b24e61d165963c5334bde9e8dc84b20debe552"}
Jul 09 16:38:53.307904 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:53.307729    2558 scope.go:117] "RemoveContainer" containerID="b93d84eb3ef0115fa87886ee125baaa6a94a1cdd3dfc8cbb5c3648a71240bea4"
Jul 09 16:38:53.308455 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:53.307970    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"
Jul 09 16:38:53.330740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:53.330710    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"]
Jul 09 16:38:53.332910 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:53.332867    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-85fe79d8-openshift-default-957b4475-7kthv"]
Jul 09 16:38:55.178035 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:38:55.177997    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" path="/var/lib/kubelet/pods/7cc7da2c-769c-4483-a124-a5c1d2dfd338/volumes"
Jul 09 16:39:15.198048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:15.198019    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:39:15.198820 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:15.198805    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:39:41.190024 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.189991    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"]
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190336    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f91a4eba-d555-466f-afe1-2764192f148f" containerName="main"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190348    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f91a4eba-d555-466f-afe1-2764192f148f" containerName="main"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190368    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190375    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190386    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fedb83e6-b529-41e3-85a4-e54b7dea3ccc" containerName="maas-api"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190391    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="fedb83e6-b529-41e3-85a4-e54b7dea3ccc" containerName="maas-api"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190399    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190405    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190461    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f91a4eba-d555-466f-afe1-2764192f148f" containerName="main"
Jul 09 16:39:41.190466 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190469    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="fedb83e6-b529-41e3-85a4-e54b7dea3ccc" containerName="maas-api"
Jul 09 16:39:41.190774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.190475    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy"
Jul 09 16:39:41.193550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.193526    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.196380 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.196348    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-36a592-openshift-default-dockercfg-6nlw8\""
Jul 09 16:39:41.206231 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.206204    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"]
Jul 09 16:39:41.300615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300572    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29400633-adc4-419c-af9f-f15cdfe82d79-istiod-ca-cert\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.300615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300621    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t5hl\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-kube-api-access-5t5hl\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.300871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300752    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-certs\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.300871 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300793    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-credential-socket\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.301022 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300908    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29400633-adc4-419c-af9f-f15cdfe82d79-istio-podinfo\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.301022 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-data\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.301022 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.300979    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-istio-token\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.301138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.301031    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-socket\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.301138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.301049    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-envoy\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.401946 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.401910    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29400633-adc4-419c-af9f-f15cdfe82d79-istio-podinfo\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.401946 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.401949    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-data\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.401973    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-istio-token\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402009    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-socket\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-envoy\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402041    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29400633-adc4-419c-af9f-f15cdfe82d79-istiod-ca-cert\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402060    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t5hl\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-kube-api-access-5t5hl\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402131    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-certs\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402154    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-credential-socket\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-data\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402569    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-socket\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402758 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402733    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-certs\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402822 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402764    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-credential-socket\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.402822 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.402780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29400633-adc4-419c-af9f-f15cdfe82d79-istiod-ca-cert\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.404537 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.404520    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29400633-adc4-419c-af9f-f15cdfe82d79-istio-podinfo\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.404630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.404613    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-envoy\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.410505 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.410480    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-istio-token\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.411007 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.410986    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t5hl\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-kube-api-access-5t5hl\") pod \"e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") " pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.504218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.504130    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:41.644152 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.644127    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"]
Jul 09 16:39:41.646180 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:39:41.646149    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29400633_adc4_419c_af9f_f15cdfe82d79.slice/crio-4d3a4c471a0c6dedf9c7deae532f967a1e866b7fc47e68f7c9e5ad7980186d25 WatchSource:0}: Error finding container 4d3a4c471a0c6dedf9c7deae532f967a1e866b7fc47e68f7c9e5ad7980186d25: Status 404 returned error can't find the container with id 4d3a4c471a0c6dedf9c7deae532f967a1e866b7fc47e68f7c9e5ad7980186d25
Jul 09 16:39:41.648365 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.648334    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:39:41.648434 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.648403    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:39:41.648480 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:41.648435    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:39:42.570911 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:42.570855    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" event={"ID":"29400633-adc4-419c-af9f-f15cdfe82d79","Type":"ContainerStarted","Data":"45afa1a995e214ce9f23232d592b3bcb194927b3a34974424af2aa5864545ac1"}
Jul 09 16:39:42.571328 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:42.570928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" event={"ID":"29400633-adc4-419c-af9f-f15cdfe82d79","Type":"ContainerStarted","Data":"4d3a4c471a0c6dedf9c7deae532f967a1e866b7fc47e68f7c9e5ad7980186d25"}
Jul 09 16:39:42.593064 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:42.593016    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podStartSLOduration=1.593000273 podStartE2EDuration="1.593000273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:39:42.590738353 +0000 UTC m=+2127.972555142" watchObservedRunningTime="2026-07-09 16:39:42.593000273 +0000 UTC m=+2127.974817061"
Jul 09 16:39:43.505072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:43.505028    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:43.510160 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:43.510128    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:43.576102 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:43.576068    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:43.577230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:43.577204    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:39:44.761406 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.761363    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"]
Jul 09 16:39:44.762021 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.762003    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cc7da2c-769c-4483-a124-a5c1d2dfd338" containerName="istio-proxy"
Jul 09 16:39:44.765624 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.765603    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.768805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.768780    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-36a592-serving-cert\""
Jul 09 16:39:44.768805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.768793    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:39:44.768805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.768793    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:39:44.770047 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.770025    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:39:44.775728 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.775703    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"]
Jul 09 16:39:44.835279 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.835242    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.835279 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.835285    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc7z6\" (UniqueName: \"kubernetes.io/projected/e12ba938-a4ca-4f02-8823-28cfd20ddc66-kube-api-access-zc7z6\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.835515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.835357    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba938-a4ca-4f02-8823-28cfd20ddc66-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.936476 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.936434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.936476 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.936482    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc7z6\" (UniqueName: \"kubernetes.io/projected/e12ba938-a4ca-4f02-8823-28cfd20ddc66-kube-api-access-zc7z6\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.936797 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.936589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba938-a4ca-4f02-8823-28cfd20ddc66-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.936923 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:39:44.936873    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-serving-cert: secret "maas-api-e2e-inf-a-36a592-serving-cert" not found
Jul 09 16:39:44.937110 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:39:44.937092    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls podName:e12ba938-a4ca-4f02-8823-28cfd20ddc66 nodeName:}" failed. No retries permitted until 2026-07-09 16:39:45.437064895 +0000 UTC m=+2130.818881660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls") pod "maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" (UID: "e12ba938-a4ca-4f02-8823-28cfd20ddc66") : secret "maas-api-e2e-inf-a-36a592-serving-cert" not found
Jul 09 16:39:44.937406 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.937387    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba938-a4ca-4f02-8823-28cfd20ddc66-openshift-service-ca\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:44.945866 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:44.945840    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc7z6\" (UniqueName: \"kubernetes.io/projected/e12ba938-a4ca-4f02-8823-28cfd20ddc66-kube-api-access-zc7z6\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:45.442218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:45.442180    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:45.445129 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:45.445096    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls\") pod \"maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:45.678811 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:45.678771    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:45.824449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:45.824415    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"]
Jul 09 16:39:45.825632 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:39:45.825599    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12ba938_a4ca_4f02_8823_28cfd20ddc66.slice/crio-f5ba9642f387ac946bb75a6ea0939145c49acb1945253003e8da422b42536efe WatchSource:0}: Error finding container f5ba9642f387ac946bb75a6ea0939145c49acb1945253003e8da422b42536efe: Status 404 returned error can't find the container with id f5ba9642f387ac946bb75a6ea0939145c49acb1945253003e8da422b42536efe
Jul 09 16:39:46.593169 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:46.593129    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" event={"ID":"e12ba938-a4ca-4f02-8823-28cfd20ddc66","Type":"ContainerStarted","Data":"c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1"}
Jul 09 16:39:46.593422 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:46.593199    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:46.593422 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:46.593224    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" event={"ID":"e12ba938-a4ca-4f02-8823-28cfd20ddc66","Type":"ContainerStarted","Data":"f5ba9642f387ac946bb75a6ea0939145c49acb1945253003e8da422b42536efe"}
Jul 09 16:39:46.611970 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:46.611913    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" podStartSLOduration=2.129045891 podStartE2EDuration="2.611898325s" totalImagesPullingTime="482.852434ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:39:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:39:46.610165466 +0000 UTC m=+2131.991982255" watchObservedRunningTime="2026-07-09 16:39:46.611898325 +0000 UTC m=+2131.993715120"
Jul 09 16:39:49.734990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.734957    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"]
Jul 09 16:39:49.739217 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.739190    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.742402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.742376    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-46b53e-openshift-default-dockercfg-tnc58\""
Jul 09 16:39:49.752957 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.752928    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"]
Jul 09 16:39:49.886364 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886317    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-data\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886364 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886367    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hkml\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-kube-api-access-6hkml\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886436    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istiod-ca-cert\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886475    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-certs\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886492    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-envoy\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886512    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-credential-socket\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-podinfo\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886558    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-socket\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.886618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.886580    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-token\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988451 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988357    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istiod-ca-cert\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988486    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-certs\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988523    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-envoy\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988648 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988637    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-credential-socket\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988691    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-podinfo\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988726    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-socket\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.988987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.988751    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-token\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989055    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-data\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989100    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hkml\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-kube-api-access-6hkml\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989239 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989217    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-certs\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989331 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989286    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istiod-ca-cert\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989441    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-data\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989643 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989619    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-socket\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.989708 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.989657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-credential-socket\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.991158 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.991139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-envoy\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.991413 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.991393    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-podinfo\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.997750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.997723    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-token\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:49.997950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:49.997929    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hkml\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-kube-api-access-6hkml\") pod \"e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") " pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:50.054463 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.054420    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:50.207551 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.207522    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"]
Jul 09 16:39:50.209212 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:39:50.209175    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7ae5f8_5be2_4910_bd6a_df172ffdf85f.slice/crio-f4839c335724b766c9d86e935c88ea4c122c6a5c57bfd02731dc19ef7c36a1d9 WatchSource:0}: Error finding container f4839c335724b766c9d86e935c88ea4c122c6a5c57bfd02731dc19ef7c36a1d9: Status 404 returned error can't find the container with id f4839c335724b766c9d86e935c88ea4c122c6a5c57bfd02731dc19ef7c36a1d9
Jul 09 16:39:50.211561 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.211520    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:39:50.211681 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.211616    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:39:50.211681 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.211662    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:39:50.617406 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.617365    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" event={"ID":"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f","Type":"ContainerStarted","Data":"63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"}
Jul 09 16:39:50.617612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.617421    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" event={"ID":"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f","Type":"ContainerStarted","Data":"f4839c335724b766c9d86e935c88ea4c122c6a5c57bfd02731dc19ef7c36a1d9"}
Jul 09 16:39:50.639707 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:50.639625    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podStartSLOduration=1.6396087910000001 podStartE2EDuration="1.639608791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:39:50.638930665 +0000 UTC m=+2136.020747453" watchObservedRunningTime="2026-07-09 16:39:50.639608791 +0000 UTC m=+2136.021425579"
Jul 09 16:39:51.055292 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:51.055191    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:51.060445 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:51.060417    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:51.623066 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:51.623034    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:51.623990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:51.623972    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:39:52.605179 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.605148    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:39:52.670800 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.670562    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"]
Jul 09 16:39:52.674641 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.674614    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.677685 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.677660    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-46b53e-serving-cert\""
Jul 09 16:39:52.686632 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.686605    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"]
Jul 09 16:39:52.815951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.815905    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301d3ef3-f36e-4ac3-8fdf-186c52d20566-maas-api-tls\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.816129 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.815978    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301d3ef3-f36e-4ac3-8fdf-186c52d20566-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.816129 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.816110    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrdl2\" (UniqueName: \"kubernetes.io/projected/301d3ef3-f36e-4ac3-8fdf-186c52d20566-kube-api-access-qrdl2\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.917475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.917375    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrdl2\" (UniqueName: \"kubernetes.io/projected/301d3ef3-f36e-4ac3-8fdf-186c52d20566-kube-api-access-qrdl2\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.917475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.917434    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301d3ef3-f36e-4ac3-8fdf-186c52d20566-maas-api-tls\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.917475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.917464    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301d3ef3-f36e-4ac3-8fdf-186c52d20566-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.918155 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.918130    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301d3ef3-f36e-4ac3-8fdf-186c52d20566-openshift-service-ca\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.920191 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.920166    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301d3ef3-f36e-4ac3-8fdf-186c52d20566-maas-api-tls\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.927297 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.927269    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrdl2\" (UniqueName: \"kubernetes.io/projected/301d3ef3-f36e-4ac3-8fdf-186c52d20566-kube-api-access-qrdl2\") pod \"maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:52.991500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:52.991458    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:53.145613 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:53.145585    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"]
Jul 09 16:39:53.635298 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:53.635259    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5" event={"ID":"301d3ef3-f36e-4ac3-8fdf-186c52d20566","Type":"ContainerStarted","Data":"972d5bf8231c277dd88f48263be5f32d1f6e086f98cf867923b69e4776c2019d"}
Jul 09 16:39:54.642449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:54.642404    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5" event={"ID":"301d3ef3-f36e-4ac3-8fdf-186c52d20566","Type":"ContainerStarted","Data":"1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58"}
Jul 09 16:39:54.642929 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:54.642478    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:39:54.659654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:54.659602    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5" podStartSLOduration=2.030047987 podStartE2EDuration="2.659586565s" totalImagesPullingTime="629.538578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:39:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:39:54.658291167 +0000 UTC m=+2140.040107958" watchObservedRunningTime="2026-07-09 16:39:54.659586565 +0000 UTC m=+2140.041403352"
Jul 09 16:39:59.618097 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:59.618046    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 16:39:59.618519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:39:59.618114    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 16:40:00.660663 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:00.660628    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:40:01.688447 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:01.688414    2558 generic.go:356] "Generic (PLEG): container finished" podID="29400633-adc4-419c-af9f-f15cdfe82d79" containerID="45afa1a995e214ce9f23232d592b3bcb194927b3a34974424af2aa5864545ac1" exitCode=0
Jul 09 16:40:01.688953 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:01.688499    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" event={"ID":"29400633-adc4-419c-af9f-f15cdfe82d79","Type":"ContainerDied","Data":"45afa1a995e214ce9f23232d592b3bcb194927b3a34974424af2aa5864545ac1"}
Jul 09 16:40:01.689017 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:01.688963    2558 scope.go:117] "RemoveContainer" containerID="45afa1a995e214ce9f23232d592b3bcb194927b3a34974424af2aa5864545ac1"
Jul 09 16:40:01.689982 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:01.689956    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:40:01.690060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:01.690046    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:40:01.690109 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:01.690083    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:40:02.695500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:02.695462    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" event={"ID":"29400633-adc4-419c-af9f-f15cdfe82d79","Type":"ContainerStarted","Data":"2384e63375992511581519a6641ea3392dba89d7507a480634d466e2bbc70659"}
Jul 09 16:40:03.505289 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:03.505248    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:40:03.505484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:03.505299    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 16:40:03.505484 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:03.505358    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 16:40:04.505093 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:04.505054    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 16:40:04.505493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:04.505110    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 16:40:05.504667 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:05.504580    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 16:40:05.504667 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:05.504641    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 16:40:06.504784 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:06.504743    2558 patch_prober.go:28] interesting pod/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body=
Jul 09 16:40:06.505276 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:06.504820    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused"
Jul 09 16:40:07.508790 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.508749    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:40:07.509297 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.509272    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:40:07.510299 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.510274    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:40:07.666098 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.666055    2558 patch_prober.go:28] interesting pod/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 16:40:07.666296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.666131    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 16:40:07.968449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.968415    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"]
Jul 09 16:40:07.971131 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.971109    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:07.974292 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.974256    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-36a592\"/\"default-dockercfg-fxwgp\""
Jul 09 16:40:07.974433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.974299    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-36a592\"/\"kube-root-ca.crt\""
Jul 09 16:40:07.974433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.974268    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-36a592\"/\"openshift-service-ca.crt\""
Jul 09 16:40:07.974433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.974257    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-36a592\"/\"test-model-36a592-kserve-self-signed-certs\""
Jul 09 16:40:07.982672 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:07.982637    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"]
Jul 09 16:40:08.074166 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.074132    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tls-certs\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.074347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.074187    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-home\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.074347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.074290    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmgqf\" (UniqueName: \"kubernetes.io/projected/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-kube-api-access-kmgqf\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.074347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.074323    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-model-cache\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.074509 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.074365    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tmp-dir\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.074509 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.074420    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-dshm\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175021 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.174988    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmgqf\" (UniqueName: \"kubernetes.io/projected/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-kube-api-access-kmgqf\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175021 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175027    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-model-cache\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175057    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tmp-dir\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175076    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-dshm\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175112    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tls-certs\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175162    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-home\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175443    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-model-cache\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175514 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175497    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tmp-dir\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.175601 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.175584    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-home\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.177646 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.177616    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-dshm\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.177939 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.177911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tls-certs\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.183226 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.183199    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmgqf\" (UniqueName: \"kubernetes.io/projected/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-kube-api-access-kmgqf\") pod \"test-model-36a592-kserve-8cb4cdd5c-phlb9\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") " pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.283149 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.283051    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:08.456070 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.456037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"]
Jul 09 16:40:08.457598 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:40:08.457564    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7b3cd2c_fb40_4f8c_b90a_8f1a3000b530.slice/crio-360f8bf652e039546b25632722423713e9725e4c85086c7d780ba422af7534fb WatchSource:0}: Error finding container 360f8bf652e039546b25632722423713e9725e4c85086c7d780ba422af7534fb: Status 404 returned error can't find the container with id 360f8bf652e039546b25632722423713e9725e4c85086c7d780ba422af7534fb
Jul 09 16:40:08.742678 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.742630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9" event={"ID":"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530","Type":"ContainerStarted","Data":"3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1"}
Jul 09 16:40:08.742678 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.742682    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9" event={"ID":"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530","Type":"ContainerStarted","Data":"360f8bf652e039546b25632722423713e9725e4c85086c7d780ba422af7534fb"}
Jul 09 16:40:08.760259 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:08.760213    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9" podStartSLOduration=1.760194222 podStartE2EDuration="1.760194222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:40:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:40:08.759092928 +0000 UTC m=+2154.140909716" watchObservedRunningTime="2026-07-09 16:40:08.760194222 +0000 UTC m=+2154.142011033"
Jul 09 16:40:09.750941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:09.750799    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerID="63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf" exitCode=0
Jul 09 16:40:09.750941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:09.750907    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" event={"ID":"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f","Type":"ContainerDied","Data":"63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"}
Jul 09 16:40:09.751487 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:09.751468    2558 scope.go:117] "RemoveContainer" containerID="63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"
Jul 09 16:40:09.752496 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:09.752467    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:40:09.752620 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:09.752550    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:40:09.752620 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:09.752612    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:40:10.758957 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:10.758916    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" event={"ID":"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f","Type":"ContainerStarted","Data":"4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4"}
Jul 09 16:40:11.054771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:11.054679    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:40:11.055033 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:11.055013    2558 patch_prober.go:28] interesting pod/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 16:40:11.055088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:11.055058    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 16:40:12.055831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:12.055795    2558 patch_prober.go:28] interesting pod/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 16:40:12.056332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:12.055857    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 16:40:13.054863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:13.054825    2558 patch_prober.go:28] interesting pod/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 16:40:13.055095 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:13.054900    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 16:40:14.055317 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:14.055273    2558 patch_prober.go:28] interesting pod/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Jul 09 16:40:14.055787 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:14.055334    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Jul 09 16:40:15.068809 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:15.068772    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:40:15.069322 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:15.069301    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:40:15.070321 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:15.070299    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:40:18.284093 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:18.284056    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:18.284632 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:18.284208    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:18.297373 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:18.297324    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:18.813021 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:18.812986    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:40:33.179351 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.179321    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"]
Jul 09 16:40:33.182220 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.182192    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.184338 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.184312    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"]
Jul 09 16:40:33.185488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.185462    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46b53e\"/\"openshift-service-ca.crt\""
Jul 09 16:40:33.185634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.185612    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46b53e\"/\"test-model-46b53e-kserve-self-signed-certs\""
Jul 09 16:40:33.185707 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.185662    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-46b53e\"/\"kube-root-ca.crt\""
Jul 09 16:40:33.185768 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.185472    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-46b53e\"/\"default-dockercfg-d7rh4\""
Jul 09 16:40:33.322793 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.322753    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-dshm\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.322793 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.322799    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d05273-6140-47da-bd7e-925cbf1bd762-tls-certs\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.323070 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.322818    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvthw\" (UniqueName: \"kubernetes.io/projected/18d05273-6140-47da-bd7e-925cbf1bd762-kube-api-access-rvthw\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.323070 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.322941    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-model-cache\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.323070 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.323061    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-home\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.323237 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.323096    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-tmp-dir\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424320 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424279    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-model-cache\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424359    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-home\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424389    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-tmp-dir\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424423    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-dshm\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424447    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d05273-6140-47da-bd7e-925cbf1bd762-tls-certs\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424468    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvthw\" (UniqueName: \"kubernetes.io/projected/18d05273-6140-47da-bd7e-925cbf1bd762-kube-api-access-rvthw\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424788 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-model-cache\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424788 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424760    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-home\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.424941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.424911    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-tmp-dir\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.426976 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.426953    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-dshm\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.427224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.427207    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d05273-6140-47da-bd7e-925cbf1bd762-tls-certs\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.433365 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.433308    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvthw\" (UniqueName: \"kubernetes.io/projected/18d05273-6140-47da-bd7e-925cbf1bd762-kube-api-access-rvthw\") pod \"test-model-46b53e-kserve-799f9744f9-69bn7\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") " pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.493549 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.493501    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:33.641164 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.641132    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"]
Jul 09 16:40:33.643350 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:40:33.643301    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod18d05273_6140_47da_bd7e_925cbf1bd762.slice/crio-69ce932026e8d6027bae9b27d5d42a346b853facff39ae95331097db0395f9ba WatchSource:0}: Error finding container 69ce932026e8d6027bae9b27d5d42a346b853facff39ae95331097db0395f9ba: Status 404 returned error can't find the container with id 69ce932026e8d6027bae9b27d5d42a346b853facff39ae95331097db0395f9ba
Jul 09 16:40:33.889622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.889581    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7" event={"ID":"18d05273-6140-47da-bd7e-925cbf1bd762","Type":"ContainerStarted","Data":"a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659"}
Jul 09 16:40:33.889622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.889636    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7" event={"ID":"18d05273-6140-47da-bd7e-925cbf1bd762","Type":"ContainerStarted","Data":"69ce932026e8d6027bae9b27d5d42a346b853facff39ae95331097db0395f9ba"}
Jul 09 16:40:33.907710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:33.907650    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7" podStartSLOduration=0.907628855 podStartE2EDuration="907.628855ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:40:33.906871453 +0000 UTC m=+2179.288688244" watchObservedRunningTime="2026-07-09 16:40:33.907628855 +0000 UTC m=+2179.289445644"
Jul 09 16:40:43.494179 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:43.494133    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:43.494753 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:43.494208    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:43.507052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:43.507021    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:43.960212 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:43.960174    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:40:55.545713 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:55.545676    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"]
Jul 09 16:40:56.587683 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:56.587642    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"]
Jul 09 16:40:57.403001 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.402954    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"]
Jul 09 16:40:57.403217 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.403190    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" podUID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" containerName="maas-api" containerID="cri-o://c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1" gracePeriod=30
Jul 09 16:40:57.665642 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.665576    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:40:57.762832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.762788    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba938-a4ca-4f02-8823-28cfd20ddc66-openshift-service-ca\") pod \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") "
Jul 09 16:40:57.763075 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.762842    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zc7z6\" (UniqueName: \"kubernetes.io/projected/e12ba938-a4ca-4f02-8823-28cfd20ddc66-kube-api-access-zc7z6\") pod \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") "
Jul 09 16:40:57.763075 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.762950    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls\") pod \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\" (UID: \"e12ba938-a4ca-4f02-8823-28cfd20ddc66\") "
Jul 09 16:40:57.763288 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.763262    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12ba938-a4ca-4f02-8823-28cfd20ddc66-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e12ba938-a4ca-4f02-8823-28cfd20ddc66" (UID: "e12ba938-a4ca-4f02-8823-28cfd20ddc66"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:40:57.765339 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.765305    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12ba938-a4ca-4f02-8823-28cfd20ddc66-kube-api-access-zc7z6" (OuterVolumeSpecName: "kube-api-access-zc7z6") pod "e12ba938-a4ca-4f02-8823-28cfd20ddc66" (UID: "e12ba938-a4ca-4f02-8823-28cfd20ddc66"). InnerVolumeSpecName "kube-api-access-zc7z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:40:57.765436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.765343    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e12ba938-a4ca-4f02-8823-28cfd20ddc66" (UID: "e12ba938-a4ca-4f02-8823-28cfd20ddc66"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:40:57.863941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.863903    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e12ba938-a4ca-4f02-8823-28cfd20ddc66-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:40:57.863941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.863935    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc7z6\" (UniqueName: \"kubernetes.io/projected/e12ba938-a4ca-4f02-8823-28cfd20ddc66-kube-api-access-zc7z6\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:40:57.863941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:57.863946    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e12ba938-a4ca-4f02-8823-28cfd20ddc66-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:40:58.027664 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.027569    2558 generic.go:356] "Generic (PLEG): container finished" podID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" containerID="c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1" exitCode=0
Jul 09 16:40:58.027664 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.027624    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"
Jul 09 16:40:58.027971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.027622    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" event={"ID":"e12ba938-a4ca-4f02-8823-28cfd20ddc66","Type":"ContainerDied","Data":"c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1"}
Jul 09 16:40:58.027971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.027734    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" event={"ID":"e12ba938-a4ca-4f02-8823-28cfd20ddc66","Type":"ContainerDied","Data":"f5ba9642f387ac946bb75a6ea0939145c49acb1945253003e8da422b42536efe"}
Jul 09 16:40:58.027971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.027763    2558 scope.go:117] "RemoveContainer" containerID="c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1"
Jul 09 16:40:58.038396 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.038378    2558 scope.go:117] "RemoveContainer" containerID="c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1"
Jul 09 16:40:58.038676 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:40:58.038654    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1\": container with ID starting with c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1 not found: ID does not exist" containerID="c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1"
Jul 09 16:40:58.038720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.038686    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1"} err="failed to get container status \"c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1\": rpc error: code = NotFound desc = could not find container \"c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1\": container with ID starting with c30fca528e09a15deea27f6dec85806c2b0fcbf14452098bc877b9b1f2dc07b1 not found: ID does not exist"
Jul 09 16:40:58.054170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.054140    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"]
Jul 09 16:40:58.056390 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:58.056368    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2"]
Jul 09 16:40:59.178903 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:40:59.178835    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" path="/var/lib/kubelet/pods/e12ba938-a4ca-4f02-8823-28cfd20ddc66/volumes"
Jul 09 16:41:00.598488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:00.598440    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-36a592-b7f7cdc6c-kz4k2" podUID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.60:8443/health\": context deadline exceeded"
Jul 09 16:41:10.566950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.566872    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9" podUID="e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" containerName="main" containerID="cri-o://3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1" gracePeriod=45
Jul 09 16:41:10.819743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.819675    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:41:10.891575 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891532    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tls-certs\") pod \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") "
Jul 09 16:41:10.891783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891596    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-home\") pod \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") "
Jul 09 16:41:10.891783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891615    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmgqf\" (UniqueName: \"kubernetes.io/projected/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-kube-api-access-kmgqf\") pod \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") "
Jul 09 16:41:10.891783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891631    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tmp-dir\") pod \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") "
Jul 09 16:41:10.891783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891658    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-dshm\") pod \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") "
Jul 09 16:41:10.891783 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891684    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-model-cache\") pod \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\" (UID: \"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530\") "
Jul 09 16:41:10.892088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891926    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-home" (OuterVolumeSpecName: "home") pod "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" (UID: "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:10.892088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.891946    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" (UID: "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:10.892088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.892049    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-model-cache" (OuterVolumeSpecName: "model-cache") pod "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" (UID: "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:10.892209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.892150    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:10.892209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.892166    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:10.892209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.892174    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:10.894185 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.894163    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-dshm" (OuterVolumeSpecName: "dshm") pod "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" (UID: "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:10.894278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.894192    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-kube-api-access-kmgqf" (OuterVolumeSpecName: "kube-api-access-kmgqf") pod "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" (UID: "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530"). InnerVolumeSpecName "kube-api-access-kmgqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:10.894278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.894243    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" (UID: "e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:41:10.993052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.993003    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:10.993052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.993047    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmgqf\" (UniqueName: \"kubernetes.io/projected/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-kube-api-access-kmgqf\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:10.993052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:10.993060    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:11.110697 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.110609    2558 generic.go:356] "Generic (PLEG): container finished" podID="e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" containerID="3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1" exitCode=0
Jul 09 16:41:11.110697 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.110675    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"
Jul 09 16:41:11.110907 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.110700    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9" event={"ID":"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530","Type":"ContainerDied","Data":"3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1"}
Jul 09 16:41:11.110907 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.110751    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9" event={"ID":"e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530","Type":"ContainerDied","Data":"360f8bf652e039546b25632722423713e9725e4c85086c7d780ba422af7534fb"}
Jul 09 16:41:11.110907 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.110768    2558 scope.go:117] "RemoveContainer" containerID="3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1"
Jul 09 16:41:11.120581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.120550    2558 scope.go:117] "RemoveContainer" containerID="3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1"
Jul 09 16:41:11.120908 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:11.120869    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1\": container with ID starting with 3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1 not found: ID does not exist" containerID="3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1"
Jul 09 16:41:11.120962 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.120921    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1"} err="failed to get container status \"3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1\": rpc error: code = NotFound desc = could not find container \"3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1\": container with ID starting with 3d18761d40c7e3bc8899cd4d2b8c619213bb1d463bfe163c07fd73d97632abe1 not found: ID does not exist"
Jul 09 16:41:11.133112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.133079    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"]
Jul 09 16:41:11.136474 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.136449    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-36a592/test-model-36a592-kserve-8cb4cdd5c-phlb9"]
Jul 09 16:41:11.178587 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.178550    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" path="/var/lib/kubelet/pods/e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530/volumes"
Jul 09 16:41:11.610348 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.610304    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7" podUID="18d05273-6140-47da-bd7e-925cbf1bd762" containerName="main" containerID="cri-o://a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659" gracePeriod=45
Jul 09 16:41:11.869472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:11.869405    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:41:12.000680 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.000639    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d05273-6140-47da-bd7e-925cbf1bd762-tls-certs\") pod \"18d05273-6140-47da-bd7e-925cbf1bd762\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") "
Jul 09 16:41:12.000906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.000699    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-tmp-dir\") pod \"18d05273-6140-47da-bd7e-925cbf1bd762\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") "
Jul 09 16:41:12.000906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.000719    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-model-cache\") pod \"18d05273-6140-47da-bd7e-925cbf1bd762\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") "
Jul 09 16:41:12.000906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.000741    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvthw\" (UniqueName: \"kubernetes.io/projected/18d05273-6140-47da-bd7e-925cbf1bd762-kube-api-access-rvthw\") pod \"18d05273-6140-47da-bd7e-925cbf1bd762\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") "
Jul 09 16:41:12.000906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.000772    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-home\") pod \"18d05273-6140-47da-bd7e-925cbf1bd762\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") "
Jul 09 16:41:12.000906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.000794    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-dshm\") pod \"18d05273-6140-47da-bd7e-925cbf1bd762\" (UID: \"18d05273-6140-47da-bd7e-925cbf1bd762\") "
Jul 09 16:41:12.001174 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.001084    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-home" (OuterVolumeSpecName: "home") pod "18d05273-6140-47da-bd7e-925cbf1bd762" (UID: "18d05273-6140-47da-bd7e-925cbf1bd762"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:12.001174 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.001093    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-model-cache" (OuterVolumeSpecName: "model-cache") pod "18d05273-6140-47da-bd7e-925cbf1bd762" (UID: "18d05273-6140-47da-bd7e-925cbf1bd762"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:12.001174 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.001107    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "18d05273-6140-47da-bd7e-925cbf1bd762" (UID: "18d05273-6140-47da-bd7e-925cbf1bd762"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:12.003258 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.003232    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d05273-6140-47da-bd7e-925cbf1bd762-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "18d05273-6140-47da-bd7e-925cbf1bd762" (UID: "18d05273-6140-47da-bd7e-925cbf1bd762"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:41:12.003375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.003287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-dshm" (OuterVolumeSpecName: "dshm") pod "18d05273-6140-47da-bd7e-925cbf1bd762" (UID: "18d05273-6140-47da-bd7e-925cbf1bd762"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:12.003375 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.003290    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d05273-6140-47da-bd7e-925cbf1bd762-kube-api-access-rvthw" (OuterVolumeSpecName: "kube-api-access-rvthw") pod "18d05273-6140-47da-bd7e-925cbf1bd762" (UID: "18d05273-6140-47da-bd7e-925cbf1bd762"). InnerVolumeSpecName "kube-api-access-rvthw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:12.102435 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.102394    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:12.102435 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.102430    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:12.102435 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.102441    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvthw\" (UniqueName: \"kubernetes.io/projected/18d05273-6140-47da-bd7e-925cbf1bd762-kube-api-access-rvthw\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:12.102686 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.102450    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:12.102686 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.102459    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/18d05273-6140-47da-bd7e-925cbf1bd762-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:12.102686 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.102469    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/18d05273-6140-47da-bd7e-925cbf1bd762-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:12.118581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.118545    2558 generic.go:356] "Generic (PLEG): container finished" podID="18d05273-6140-47da-bd7e-925cbf1bd762" containerID="a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659" exitCode=0
Jul 09 16:41:12.118766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.118599    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"
Jul 09 16:41:12.118766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.118598    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7" event={"ID":"18d05273-6140-47da-bd7e-925cbf1bd762","Type":"ContainerDied","Data":"a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659"}
Jul 09 16:41:12.118766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.118716    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7" event={"ID":"18d05273-6140-47da-bd7e-925cbf1bd762","Type":"ContainerDied","Data":"69ce932026e8d6027bae9b27d5d42a346b853facff39ae95331097db0395f9ba"}
Jul 09 16:41:12.118766 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.118741    2558 scope.go:117] "RemoveContainer" containerID="a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659"
Jul 09 16:41:12.129399 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.129381    2558 scope.go:117] "RemoveContainer" containerID="a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659"
Jul 09 16:41:12.129704 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:12.129684    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659\": container with ID starting with a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659 not found: ID does not exist" containerID="a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659"
Jul 09 16:41:12.129773 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.129713    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659"} err="failed to get container status \"a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659\": rpc error: code = NotFound desc = could not find container \"a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659\": container with ID starting with a937ed484b8866cc1925cd0e9b3934a129dff173f129d9010e32c4509f76e659 not found: ID does not exist"
Jul 09 16:41:12.140987 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.140950    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"]
Jul 09 16:41:12.143156 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:12.143132    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-46b53e/test-model-46b53e-kserve-799f9744f9-69bn7"]
Jul 09 16:41:13.177641 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:13.177604    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18d05273-6140-47da-bd7e-925cbf1bd762" path="/var/lib/kubelet/pods/18d05273-6140-47da-bd7e-925cbf1bd762/volumes"
Jul 09 16:41:39.180023 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:39.179941    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"]
Jul 09 16:41:39.180450 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:39.180205    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy" containerID="cri-o://2384e63375992511581519a6641ea3392dba89d7507a480634d466e2bbc70659" gracePeriod=30
Jul 09 16:41:40.396639 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.396601    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"]
Jul 09 16:41:40.397090 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.396813    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5" podUID="301d3ef3-f36e-4ac3-8fdf-186c52d20566" containerName="maas-api" containerID="cri-o://1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58" gracePeriod=30
Jul 09 16:41:40.653248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.653182    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:41:40.773330 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.773268    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrdl2\" (UniqueName: \"kubernetes.io/projected/301d3ef3-f36e-4ac3-8fdf-186c52d20566-kube-api-access-qrdl2\") pod \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") "
Jul 09 16:41:40.773518 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.773351    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301d3ef3-f36e-4ac3-8fdf-186c52d20566-openshift-service-ca\") pod \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") "
Jul 09 16:41:40.773518 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.773377    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301d3ef3-f36e-4ac3-8fdf-186c52d20566-maas-api-tls\") pod \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\" (UID: \"301d3ef3-f36e-4ac3-8fdf-186c52d20566\") "
Jul 09 16:41:40.773758 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.773733    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301d3ef3-f36e-4ac3-8fdf-186c52d20566-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "301d3ef3-f36e-4ac3-8fdf-186c52d20566" (UID: "301d3ef3-f36e-4ac3-8fdf-186c52d20566"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:41:40.775618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.775585    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301d3ef3-f36e-4ac3-8fdf-186c52d20566-kube-api-access-qrdl2" (OuterVolumeSpecName: "kube-api-access-qrdl2") pod "301d3ef3-f36e-4ac3-8fdf-186c52d20566" (UID: "301d3ef3-f36e-4ac3-8fdf-186c52d20566"). InnerVolumeSpecName "kube-api-access-qrdl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:40.775618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.775598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/301d3ef3-f36e-4ac3-8fdf-186c52d20566-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "301d3ef3-f36e-4ac3-8fdf-186c52d20566" (UID: "301d3ef3-f36e-4ac3-8fdf-186c52d20566"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:41:40.874803 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.874765    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrdl2\" (UniqueName: \"kubernetes.io/projected/301d3ef3-f36e-4ac3-8fdf-186c52d20566-kube-api-access-qrdl2\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:40.874803 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.874796    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/301d3ef3-f36e-4ac3-8fdf-186c52d20566-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:40.874803 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:40.874806    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/301d3ef3-f36e-4ac3-8fdf-186c52d20566-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:41.290200 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.290165    2558 generic.go:356] "Generic (PLEG): container finished" podID="301d3ef3-f36e-4ac3-8fdf-186c52d20566" containerID="1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58" exitCode=0
Jul 09 16:41:41.290394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.290228    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5" event={"ID":"301d3ef3-f36e-4ac3-8fdf-186c52d20566","Type":"ContainerDied","Data":"1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58"}
Jul 09 16:41:41.290394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.290236    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"
Jul 09 16:41:41.290394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.290269    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5" event={"ID":"301d3ef3-f36e-4ac3-8fdf-186c52d20566","Type":"ContainerDied","Data":"972d5bf8231c277dd88f48263be5f32d1f6e086f98cf867923b69e4776c2019d"}
Jul 09 16:41:41.290394 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.290289    2558 scope.go:117] "RemoveContainer" containerID="1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58"
Jul 09 16:41:41.299659 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.299635    2558 scope.go:117] "RemoveContainer" containerID="1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58"
Jul 09 16:41:41.299980 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:41.299949    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58\": container with ID starting with 1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58 not found: ID does not exist" containerID="1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58"
Jul 09 16:41:41.300092 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.299986    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58"} err="failed to get container status \"1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58\": rpc error: code = NotFound desc = could not find container \"1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58\": container with ID starting with 1fcb8d7c9ea3b36216d317924cc58c3b65a84e3ec98c1e42c72de015e9a09e58 not found: ID does not exist"
Jul 09 16:41:41.308674 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.308641    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"]
Jul 09 16:41:41.312101 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:41.312075    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-46b53e-dd9c4647-bbtr5"]
Jul 09 16:41:43.177809 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:43.177774    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="301d3ef3-f36e-4ac3-8fdf-186c52d20566" path="/var/lib/kubelet/pods/301d3ef3-f36e-4ac3-8fdf-186c52d20566/volumes"
Jul 09 16:41:44.312090 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.312044    2558 generic.go:356] "Generic (PLEG): container finished" podID="29400633-adc4-419c-af9f-f15cdfe82d79" containerID="2384e63375992511581519a6641ea3392dba89d7507a480634d466e2bbc70659" exitCode=0
Jul 09 16:41:44.312540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.312099    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" event={"ID":"29400633-adc4-419c-af9f-f15cdfe82d79","Type":"ContainerDied","Data":"2384e63375992511581519a6641ea3392dba89d7507a480634d466e2bbc70659"}
Jul 09 16:41:44.312540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.312188    2558 scope.go:117] "RemoveContainer" containerID="45afa1a995e214ce9f23232d592b3bcb194927b3a34974424af2aa5864545ac1"
Jul 09 16:41:44.446524 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.446455    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:41:44.501818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.501789    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5t5hl\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-kube-api-access-5t5hl\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.501818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.501825    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-certs\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.501854    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29400633-adc4-419c-af9f-f15cdfe82d79-istiod-ca-cert\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.501874    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-credential-socket\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.501929    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-socket\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.501972    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-istio-token\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502012    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-data\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502040    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29400633-adc4-419c-af9f-f15cdfe82d79-istio-podinfo\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502091    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-envoy\") pod \"29400633-adc4-419c-af9f-f15cdfe82d79\" (UID: \"29400633-adc4-419c-af9f-f15cdfe82d79\") "
Jul 09 16:41:44.502372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502154    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:44.502372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502273    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:44.502372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502287    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:44.502552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502423    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.502552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502427    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29400633-adc4-419c-af9f-f15cdfe82d79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:41:44.502552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502442    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.502552 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502455    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.502951 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.502929    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-data" (OuterVolumeSpecName: "istio-data") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:44.504224 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.504197    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-istio-token" (OuterVolumeSpecName: "istio-token") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:44.504462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.504442    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:44.504674 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.504655    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29400633-adc4-419c-af9f-f15cdfe82d79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:41:44.504721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.504672    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-kube-api-access-5t5hl" (OuterVolumeSpecName: "kube-api-access-5t5hl") pod "29400633-adc4-419c-af9f-f15cdfe82d79" (UID: "29400633-adc4-419c-af9f-f15cdfe82d79"). InnerVolumeSpecName "kube-api-access-5t5hl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:44.603402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.603366    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29400633-adc4-419c-af9f-f15cdfe82d79-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.603402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.603396    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.603402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.603405    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.603402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.603412    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29400633-adc4-419c-af9f-f15cdfe82d79-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.603676 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.603423    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29400633-adc4-419c-af9f-f15cdfe82d79-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:44.603676 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:44.603431    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t5hl\" (UniqueName: \"kubernetes.io/projected/29400633-adc4-419c-af9f-f15cdfe82d79-kube-api-access-5t5hl\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:45.319493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.319410    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"
Jul 09 16:41:45.319493 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.319445    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr" event={"ID":"29400633-adc4-419c-af9f-f15cdfe82d79","Type":"ContainerDied","Data":"4d3a4c471a0c6dedf9c7deae532f967a1e866b7fc47e68f7c9e5ad7980186d25"}
Jul 09 16:41:45.320058 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.319507    2558 scope.go:117] "RemoveContainer" containerID="2384e63375992511581519a6641ea3392dba89d7507a480634d466e2bbc70659"
Jul 09 16:41:45.338912 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.338863    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"]
Jul 09 16:41:45.342521 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.342492    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-36a592-openshift-default-6c74b7f449-6xwnr"]
Jul 09 16:41:45.918150 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.918117    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"]
Jul 09 16:41:45.918399 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:45.918368    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy" containerID="cri-o://4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4" gracePeriod=30
Jul 09 16:41:46.800103 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800062    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"]
Jul 09 16:41:46.800479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800445    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" containerName="main"
Jul 09 16:41:46.800479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800458    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" containerName="main"
Jul 09 16:41:46.800479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800473    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy"
Jul 09 16:41:46.800479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800478    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800489    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18d05273-6140-47da-bd7e-925cbf1bd762" containerName="main"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800495    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="18d05273-6140-47da-bd7e-925cbf1bd762" containerName="main"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800508    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="301d3ef3-f36e-4ac3-8fdf-186c52d20566" containerName="maas-api"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800515    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="301d3ef3-f36e-4ac3-8fdf-186c52d20566" containerName="maas-api"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800523    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" containerName="maas-api"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800528    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" containerName="maas-api"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800586    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b3cd2c-fb40-4f8c-b90a-8f1a3000b530" containerName="main"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800593    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="18d05273-6140-47da-bd7e-925cbf1bd762" containerName="main"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800600    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800606    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800613    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="301d3ef3-f36e-4ac3-8fdf-186c52d20566" containerName="maas-api"
Jul 09 16:41:46.800623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.800619    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="e12ba938-a4ca-4f02-8823-28cfd20ddc66" containerName="maas-api"
Jul 09 16:41:46.805145 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.805124    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.808178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.808153    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b7d28b-openshift-default-dockercfg-8cvrd\""
Jul 09 16:41:46.818825 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.818798    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"]
Jul 09 16:41:46.822524 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-token\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822657 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822558    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-certs\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822657 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822631    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-envoy\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822748 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822668    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istiod-ca-cert\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822748 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822712    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk5m8\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-kube-api-access-nk5m8\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822826 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822764    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-podinfo\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822826 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822819    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-credential-socket\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822932 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822842    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-data\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.822932 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.822868    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-socket\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.923670 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.923630    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-envoy\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.923670 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.923680    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istiod-ca-cert\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.923968 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.923802    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk5m8\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-kube-api-access-nk5m8\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924034 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.923989    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-podinfo\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924124 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924105    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-credential-socket\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924139    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-data\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924178 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924171    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-socket\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924282 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924239    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-token\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924290    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-certs\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924480 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924454    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istiod-ca-cert\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924549 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924482    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-data\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924643 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924620    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-socket\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924819 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924722    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-certs\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.924819 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.924759    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-credential-socket\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.926472 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.926444    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-envoy\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.926745 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.926727    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-podinfo\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.932491 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.932466    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-token\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:46.932596 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:46.932535    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk5m8\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-kube-api-access-nk5m8\") pod \"e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") " pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:47.117234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.117187    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:47.183132 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.183092    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" path="/var/lib/kubelet/pods/29400633-adc4-419c-af9f-f15cdfe82d79/volumes"
Jul 09 16:41:47.267044 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.267020    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"]
Jul 09 16:41:47.269486 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:41:47.269460    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4681cc3d_30d4_4ee7_b30f_bef3a925b723.slice/crio-94865c36d111193552c833db2598d900305f6772432fa82e935a5f7569441cb0 WatchSource:0}: Error finding container 94865c36d111193552c833db2598d900305f6772432fa82e935a5f7569441cb0: Status 404 returned error can't find the container with id 94865c36d111193552c833db2598d900305f6772432fa82e935a5f7569441cb0
Jul 09 16:41:47.271681 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.271637    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:41:47.271763 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.271734    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:41:47.271806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.271774    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:41:47.333588 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:47.333563    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" event={"ID":"4681cc3d-30d4-4ee7-b30f-bef3a925b723","Type":"ContainerStarted","Data":"94865c36d111193552c833db2598d900305f6772432fa82e935a5f7569441cb0"}
Jul 09 16:41:48.340419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:48.340381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" event={"ID":"4681cc3d-30d4-4ee7-b30f-bef3a925b723","Type":"ContainerStarted","Data":"d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"}
Jul 09 16:41:48.363467 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:48.363417    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podStartSLOduration=2.3634003 podStartE2EDuration="2.3634003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:48.361400784 +0000 UTC m=+2253.743217572" watchObservedRunningTime="2026-07-09 16:41:48.3634003 +0000 UTC m=+2253.745217086"
Jul 09 16:41:49.117820 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:49.117777    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:49.123011 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:49.122980    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:49.345767 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:49.345737    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:49.346776 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:49.346757    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:41:50.443730 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.443693    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"]
Jul 09 16:41:50.444225 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.444063    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy"
Jul 09 16:41:50.444225 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.444074    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="29400633-adc4-419c-af9f-f15cdfe82d79" containerName="istio-proxy"
Jul 09 16:41:50.447100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.447077    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.450132 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.450106    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b7d28b-serving-cert\""
Jul 09 16:41:50.450281 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.450143    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:41:50.450281 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.450220    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:41:50.451499 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.451482    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:41:50.462156 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.462129    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"]
Jul 09 16:41:50.555198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.555158    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh5q7\" (UniqueName: \"kubernetes.io/projected/821df56f-e1be-43f6-b110-e367aaf64b53-kube-api-access-lh5q7\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.555382 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.555219    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.555476 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.555387    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821df56f-e1be-43f6-b110-e367aaf64b53-openshift-service-ca\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.656296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.656256    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821df56f-e1be-43f6-b110-e367aaf64b53-openshift-service-ca\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.656511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.656309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh5q7\" (UniqueName: \"kubernetes.io/projected/821df56f-e1be-43f6-b110-e367aaf64b53-kube-api-access-lh5q7\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.656511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.656354    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.656511 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:50.656465    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-serving-cert: secret "maas-api-e2e-api-a-b7d28b-serving-cert" not found
Jul 09 16:41:50.656680 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:50.656540    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls podName:821df56f-e1be-43f6-b110-e367aaf64b53 nodeName:}" failed. No retries permitted until 2026-07-09 16:41:51.156519759 +0000 UTC m=+2256.538336538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls") pod "maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" (UID: "821df56f-e1be-43f6-b110-e367aaf64b53") : secret "maas-api-e2e-api-a-b7d28b-serving-cert" not found
Jul 09 16:41:50.657036 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.657008    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821df56f-e1be-43f6-b110-e367aaf64b53-openshift-service-ca\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:50.673360 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:50.673334    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh5q7\" (UniqueName: \"kubernetes.io/projected/821df56f-e1be-43f6-b110-e367aaf64b53-kube-api-access-lh5q7\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:51.161777 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.161745    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:51.161977 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:51.161954    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-serving-cert: secret "maas-api-e2e-api-a-b7d28b-serving-cert" not found
Jul 09 16:41:51.162065 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:51.162052    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls podName:821df56f-e1be-43f6-b110-e367aaf64b53 nodeName:}" failed. No retries permitted until 2026-07-09 16:41:52.162029721 +0000 UTC m=+2257.543846503 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls") pod "maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" (UID: "821df56f-e1be-43f6-b110-e367aaf64b53") : secret "maas-api-e2e-api-a-b7d28b-serving-cert" not found
Jul 09 16:41:51.175718 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.175689    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:41:51.262325 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262228    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-token\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262325 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262287    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hkml\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-kube-api-access-6hkml\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262325 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262325    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istiod-ca-cert\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262364    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-certs\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262397    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-podinfo\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262419    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-credential-socket\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262442    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-socket\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262478    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-envoy\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262512    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-data\") pod \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\" (UID: \"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f\") "
Jul 09 16:41:51.262906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262679    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:51.262906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262778    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:41:51.263037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262855    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:51.263037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262945    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.263037 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.262988    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:51.263798 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.263776    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-data" (OuterVolumeSpecName: "istio-data") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:51.264770 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.264745    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-kube-api-access-6hkml" (OuterVolumeSpecName: "kube-api-access-6hkml") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "kube-api-access-6hkml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:51.264863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.264787    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:41:51.264863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.264801    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-token" (OuterVolumeSpecName: "istio-token") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:41:51.265114 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.265096    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" (UID: "3d7ae5f8-5be2-4910-bd6a-df172ffdf85f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:41:51.357595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.357559    2558 generic.go:356] "Generic (PLEG): container finished" podID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerID="4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4" exitCode=0
Jul 09 16:41:51.357740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.357610    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"
Jul 09 16:41:51.357740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.357641    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" event={"ID":"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f","Type":"ContainerDied","Data":"4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4"}
Jul 09 16:41:51.357740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.357699    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq" event={"ID":"3d7ae5f8-5be2-4910-bd6a-df172ffdf85f","Type":"ContainerDied","Data":"f4839c335724b766c9d86e935c88ea4c122c6a5c57bfd02731dc19ef7c36a1d9"}
Jul 09 16:41:51.357740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.357729    2558 scope.go:117] "RemoveContainer" containerID="4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4"
Jul 09 16:41:51.363346 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363323    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363346 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363346    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363358    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363371    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363381    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363392    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363405    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hkml\" (UniqueName: \"kubernetes.io/projected/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-kube-api-access-6hkml\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.363548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.363417    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:41:51.367362 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.367345    2558 scope.go:117] "RemoveContainer" containerID="63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"
Jul 09 16:41:51.377166 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.377098    2558 scope.go:117] "RemoveContainer" containerID="4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4"
Jul 09 16:41:51.377547 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:51.377517    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4\": container with ID starting with 4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4 not found: ID does not exist" containerID="4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4"
Jul 09 16:41:51.377832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.377559    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4"} err="failed to get container status \"4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4\": rpc error: code = NotFound desc = could not find container \"4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4\": container with ID starting with 4d742b47454ec5d812a500c3cb1d79bf48f2d8ab530ea933ec60a2e28b1bd3e4 not found: ID does not exist"
Jul 09 16:41:51.377832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.377586    2558 scope.go:117] "RemoveContainer" containerID="63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"
Jul 09 16:41:51.378221 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:51.378199    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf\": container with ID starting with 63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf not found: ID does not exist" containerID="63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"
Jul 09 16:41:51.378298 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.378229    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf"} err="failed to get container status \"63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf\": rpc error: code = NotFound desc = could not find container \"63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf\": container with ID starting with 63e7bf61b32c09f4afcd895502cc04fbfaf7b4eaf075ba737b827fd789f7fedf not found: ID does not exist"
Jul 09 16:41:51.382115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.382091    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"]
Jul 09 16:41:51.385721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:51.385696    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46b53e-openshift-default-cd8c5d649-c54gq"]
Jul 09 16:41:52.173424 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:52.173381    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:52.175977 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:52.175958    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") pod \"maas-api-e2e-api-a-b7d28b-675fff957-rkqqh\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:52.258119 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:52.258080    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:52.403774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:52.403745    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"]
Jul 09 16:41:52.405679 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:41:52.405647    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821df56f_e1be_43f6_b110_e367aaf64b53.slice/crio-99d9b8de0c11de15df69e944f97a47a11edfc913629315753a12c87648f0cdf6 WatchSource:0}: Error finding container 99d9b8de0c11de15df69e944f97a47a11edfc913629315753a12c87648f0cdf6: Status 404 returned error can't find the container with id 99d9b8de0c11de15df69e944f97a47a11edfc913629315753a12c87648f0cdf6
Jul 09 16:41:52.407177 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:52.407155    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:41:53.177874 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:53.177792    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" path="/var/lib/kubelet/pods/3d7ae5f8-5be2-4910-bd6a-df172ffdf85f/volumes"
Jul 09 16:41:53.376476 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:53.376442    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" event={"ID":"821df56f-e1be-43f6-b110-e367aaf64b53","Type":"ContainerStarted","Data":"193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78"}
Jul 09 16:41:53.376630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:53.376491    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" event={"ID":"821df56f-e1be-43f6-b110-e367aaf64b53","Type":"ContainerStarted","Data":"99d9b8de0c11de15df69e944f97a47a11edfc913629315753a12c87648f0cdf6"}
Jul 09 16:41:53.376630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:53.376532    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:53.394413 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:53.394364    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" podStartSLOduration=2.983834103 podStartE2EDuration="3.394347075s" totalImagesPullingTime="410.512972ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:53.393570255 +0000 UTC m=+2258.775387042" watchObservedRunningTime="2026-07-09 16:41:53.394347075 +0000 UTC m=+2258.776163862"
Jul 09 16:41:55.553988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.553376    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"]
Jul 09 16:41:55.553988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.553938    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy"
Jul 09 16:41:55.553988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.553957    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy"
Jul 09 16:41:55.553988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.553968    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy"
Jul 09 16:41:55.553988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.553976    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy"
Jul 09 16:41:55.554533 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.554049    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy"
Jul 09 16:41:55.554533 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.554061    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d7ae5f8-5be2-4910-bd6a-df172ffdf85f" containerName="istio-proxy"
Jul 09 16:41:55.557710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.557686    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.560750 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.560727    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0985f7-openshift-default-dockercfg-7q4vk\""
Jul 09 16:41:55.569480 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.569453    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"]
Jul 09 16:41:55.607561 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607521    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-credential-socket\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.607775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607571    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-envoy\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.607775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607598    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-istio-token\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.607775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607669    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a832e553-8d47-4d15-9fab-098faccf2b16-istio-podinfo\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.607775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607705    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-socket\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.608063 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607796    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-data\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.608063 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607854    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5mz7\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-kube-api-access-k5mz7\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.608063 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607926    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-certs\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.608063 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.607969    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a832e553-8d47-4d15-9fab-098faccf2b16-istiod-ca-cert\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709091 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709050    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5mz7\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-kube-api-access-k5mz7\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709103    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-certs\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709315 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a832e553-8d47-4d15-9fab-098faccf2b16-istiod-ca-cert\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709381 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709364    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-credential-socket\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709400    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-envoy\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709427    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-istio-token\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709478    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-certs\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709491    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a832e553-8d47-4d15-9fab-098faccf2b16-istio-podinfo\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709543    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-socket\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709693 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709589    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-data\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709749 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709708    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-credential-socket\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.709984 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.709963    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-data\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.710204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.710004    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-socket\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.710204 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.710013    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a832e553-8d47-4d15-9fab-098faccf2b16-istiod-ca-cert\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.711932 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.711901    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-envoy\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.712749 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.712725    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a832e553-8d47-4d15-9fab-098faccf2b16-istio-podinfo\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.717555 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.717531    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5mz7\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-kube-api-access-k5mz7\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.717650 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.717606    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-istio-token\") pod \"e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") " pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:55.869130 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:55.869086    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:56.017371 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.017345    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"]
Jul 09 16:41:56.019768 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:41:56.019739    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda832e553_8d47_4d15_9fab_098faccf2b16.slice/crio-3b0baef53337453b67e3e111bee774185f11791be8285219e7491b5c9080ae18 WatchSource:0}: Error finding container 3b0baef53337453b67e3e111bee774185f11791be8285219e7491b5c9080ae18: Status 404 returned error can't find the container with id 3b0baef53337453b67e3e111bee774185f11791be8285219e7491b5c9080ae18
Jul 09 16:41:56.022036 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.022005    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:41:56.022112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.022076    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:41:56.022153 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.022123    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:41:56.393706 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.393670    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" event={"ID":"a832e553-8d47-4d15-9fab-098faccf2b16","Type":"ContainerStarted","Data":"6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"}
Jul 09 16:41:56.393864 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.393723    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" event={"ID":"a832e553-8d47-4d15-9fab-098faccf2b16","Type":"ContainerStarted","Data":"3b0baef53337453b67e3e111bee774185f11791be8285219e7491b5c9080ae18"}
Jul 09 16:41:56.417831 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.417777    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podStartSLOduration=1.4177605070000001 podStartE2EDuration="1.417760507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:41:56.415046051 +0000 UTC m=+2261.796862850" watchObservedRunningTime="2026-07-09 16:41:56.417760507 +0000 UTC m=+2261.799577293"
Jul 09 16:41:56.869382 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:56.869344    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:57.874969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:57.874934    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:58.405338 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.405307    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:58.406469 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.406444    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:41:58.426016 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.421219    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"]
Jul 09 16:41:58.437442 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.437411    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.439502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.439441    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"]
Jul 09 16:41:58.443721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.443690    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-0985f7-serving-cert\""
Jul 09 16:41:58.537117 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.537071    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba935b2-c464-4b74-814d-35e3eca4204c-openshift-service-ca\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.537329 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.537265    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdvt2\" (UniqueName: \"kubernetes.io/projected/8ba935b2-c464-4b74-814d-35e3eca4204c-kube-api-access-tdvt2\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.537498 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.537471    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.638870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.638831    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdvt2\" (UniqueName: \"kubernetes.io/projected/8ba935b2-c464-4b74-814d-35e3eca4204c-kube-api-access-tdvt2\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.639108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.638920    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.639108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.639006    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba935b2-c464-4b74-814d-35e3eca4204c-openshift-service-ca\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.639108 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:58.639087    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-serving-cert: secret "maas-api-e2e-api-b-0985f7-serving-cert" not found
Jul 09 16:41:58.639258 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:41:58.639165    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls podName:8ba935b2-c464-4b74-814d-35e3eca4204c nodeName:}" failed. No retries permitted until 2026-07-09 16:41:59.139145088 +0000 UTC m=+2264.520961858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls") pod "maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" (UID: "8ba935b2-c464-4b74-814d-35e3eca4204c") : secret "maas-api-e2e-api-b-0985f7-serving-cert" not found
Jul 09 16:41:58.639805 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.639780    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba935b2-c464-4b74-814d-35e3eca4204c-openshift-service-ca\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:58.650306 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:58.650267    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdvt2\" (UniqueName: \"kubernetes.io/projected/8ba935b2-c464-4b74-814d-35e3eca4204c-kube-api-access-tdvt2\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:59.144910 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:59.144261    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:59.148347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:59.148314    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls\") pod \"maas-api-e2e-api-b-0985f7-78445c5f44-slcb8\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:59.352431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:59.352392    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:41:59.387042 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:59.387017    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:41:59.563956 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:41:59.563072    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"]
Jul 09 16:41:59.564454 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:41:59.564431    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba935b2_c464_4b74_814d_35e3eca4204c.slice/crio-e3bd6d0e02cb3e8e8fd54d244572b7dc638f3535fdc8754e0744af12b6905451 WatchSource:0}: Error finding container e3bd6d0e02cb3e8e8fd54d244572b7dc638f3535fdc8754e0744af12b6905451: Status 404 returned error can't find the container with id e3bd6d0e02cb3e8e8fd54d244572b7dc638f3535fdc8754e0744af12b6905451
Jul 09 16:42:00.429954 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:00.429921    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" event={"ID":"8ba935b2-c464-4b74-814d-35e3eca4204c","Type":"ContainerStarted","Data":"096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707"}
Jul 09 16:42:00.430488 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:00.430463    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" event={"ID":"8ba935b2-c464-4b74-814d-35e3eca4204c","Type":"ContainerStarted","Data":"e3bd6d0e02cb3e8e8fd54d244572b7dc638f3535fdc8754e0744af12b6905451"}
Jul 09 16:42:00.430631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:00.430618    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:42:00.449240 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:00.449183    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" podStartSLOduration=2.055353491 podStartE2EDuration="2.449165614s" totalImagesPullingTime="393.812123ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:42:00.448056961 +0000 UTC m=+2265.829873747" watchObservedRunningTime="2026-07-09 16:42:00.449165614 +0000 UTC m=+2265.830982400"
Jul 09 16:42:05.388100 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:05.388055    2558 patch_prober.go:28] interesting pod/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 16:42:05.388658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:05.388129    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 16:42:06.468078 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:06.468045    2558 generic.go:356] "Generic (PLEG): container finished" podID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerID="d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6" exitCode=0
Jul 09 16:42:06.468516 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:06.468122    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" event={"ID":"4681cc3d-30d4-4ee7-b30f-bef3a925b723","Type":"ContainerDied","Data":"d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"}
Jul 09 16:42:06.468595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:06.468577    2558 scope.go:117] "RemoveContainer" containerID="d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"
Jul 09 16:42:06.469578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:06.469546    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:06.469671 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:06.469618    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:06.469671 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:06.469648    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:07.447920 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:07.447858    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:42:07.475292 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:07.475245    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" event={"ID":"4681cc3d-30d4-4ee7-b30f-bef3a925b723","Type":"ContainerStarted","Data":"6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5"}
Jul 09 16:42:08.117850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:08.117811    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:42:08.118311 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:08.118194    2558 patch_prober.go:28] interesting pod/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 16:42:08.118487 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:08.118465    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 16:42:09.117985 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:09.117948    2558 patch_prober.go:28] interesting pod/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 16:42:09.118530 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:09.118499    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 16:42:10.118839 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:10.118792    2558 patch_prober.go:28] interesting pod/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 16:42:10.119357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:10.118863    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 16:42:11.117731 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:11.117690    2558 patch_prober.go:28] interesting pod/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Jul 09 16:42:11.118107 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:11.118069    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Jul 09 16:42:11.812248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:11.812210    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"]
Jul 09 16:42:11.813045 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:11.813013    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" podUID="821df56f-e1be-43f6-b110-e367aaf64b53" containerName="maas-api" containerID="cri-o://193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78" gracePeriod=30
Jul 09 16:42:12.093630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.093598    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:42:12.132246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.132212    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:42:12.133715 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.133691    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:42:12.135125 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.135106    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:42:12.172993 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.172958    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821df56f-e1be-43f6-b110-e367aaf64b53-openshift-service-ca\") pod \"821df56f-e1be-43f6-b110-e367aaf64b53\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") "
Jul 09 16:42:12.173189 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.173013    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") pod \"821df56f-e1be-43f6-b110-e367aaf64b53\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") "
Jul 09 16:42:12.173189 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.173037    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh5q7\" (UniqueName: \"kubernetes.io/projected/821df56f-e1be-43f6-b110-e367aaf64b53-kube-api-access-lh5q7\") pod \"821df56f-e1be-43f6-b110-e367aaf64b53\" (UID: \"821df56f-e1be-43f6-b110-e367aaf64b53\") "
Jul 09 16:42:12.173305 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.173269    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821df56f-e1be-43f6-b110-e367aaf64b53-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "821df56f-e1be-43f6-b110-e367aaf64b53" (UID: "821df56f-e1be-43f6-b110-e367aaf64b53"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:42:12.173476 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.173450    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/821df56f-e1be-43f6-b110-e367aaf64b53-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:12.175456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.175428    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "821df56f-e1be-43f6-b110-e367aaf64b53" (UID: "821df56f-e1be-43f6-b110-e367aaf64b53"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:42:12.175646 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.175456    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821df56f-e1be-43f6-b110-e367aaf64b53-kube-api-access-lh5q7" (OuterVolumeSpecName: "kube-api-access-lh5q7") pod "821df56f-e1be-43f6-b110-e367aaf64b53" (UID: "821df56f-e1be-43f6-b110-e367aaf64b53"). InnerVolumeSpecName "kube-api-access-lh5q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:42:12.274141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.274110    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/821df56f-e1be-43f6-b110-e367aaf64b53-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:12.274141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.274141    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh5q7\" (UniqueName: \"kubernetes.io/projected/821df56f-e1be-43f6-b110-e367aaf64b53-kube-api-access-lh5q7\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:12.505652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.505560    2558 generic.go:356] "Generic (PLEG): container finished" podID="821df56f-e1be-43f6-b110-e367aaf64b53" containerID="193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78" exitCode=0
Jul 09 16:42:12.505652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.505609    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"
Jul 09 16:42:12.505652 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.505637    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" event={"ID":"821df56f-e1be-43f6-b110-e367aaf64b53","Type":"ContainerDied","Data":"193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78"}
Jul 09 16:42:12.505950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.505686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh" event={"ID":"821df56f-e1be-43f6-b110-e367aaf64b53","Type":"ContainerDied","Data":"99d9b8de0c11de15df69e944f97a47a11edfc913629315753a12c87648f0cdf6"}
Jul 09 16:42:12.505950 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.505707    2558 scope.go:117] "RemoveContainer" containerID="193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78"
Jul 09 16:42:12.522248 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.522229    2558 scope.go:117] "RemoveContainer" containerID="193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78"
Jul 09 16:42:12.522563 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:12.522542    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78\": container with ID starting with 193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78 not found: ID does not exist" containerID="193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78"
Jul 09 16:42:12.522668 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.522569    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78"} err="failed to get container status \"193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78\": rpc error: code = NotFound desc = could not find container \"193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78\": container with ID starting with 193c64aa73fee012f1d8d0ca408e9af70449cd734f7ecc83fb18eed3b8c3cb78 not found: ID does not exist"
Jul 09 16:42:12.531499 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.531458    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"]
Jul 09 16:42:12.533780 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:12.533757    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b7d28b-675fff957-rkqqh"]
Jul 09 16:42:13.179522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:13.179483    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="821df56f-e1be-43f6-b110-e367aaf64b53" path="/var/lib/kubelet/pods/821df56f-e1be-43f6-b110-e367aaf64b53/volumes"
Jul 09 16:42:14.448048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:14.448003    2558 patch_prober.go:28] interesting pod/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 16:42:14.448450 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:14.448090    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 16:42:15.524446 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:15.524410    2558 generic.go:356] "Generic (PLEG): container finished" podID="a832e553-8d47-4d15-9fab-098faccf2b16" containerID="6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88" exitCode=0
Jul 09 16:42:15.524901 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:15.524489    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" event={"ID":"a832e553-8d47-4d15-9fab-098faccf2b16","Type":"ContainerDied","Data":"6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"}
Jul 09 16:42:15.525077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:15.525060    2558 scope.go:117] "RemoveContainer" containerID="6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"
Jul 09 16:42:15.526174 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:15.526147    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:15.526237 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:15.526214    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:15.526273 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:15.526261    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:16.532240 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:16.532206    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" event={"ID":"a832e553-8d47-4d15-9fab-098faccf2b16","Type":"ContainerStarted","Data":"7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310"}
Jul 09 16:42:16.870104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:16.870067    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:42:16.870451 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:16.870340    2558 patch_prober.go:28] interesting pod/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 16:42:16.870570 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:16.870488    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 16:42:17.474062 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:17.474028    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"]
Jul 09 16:42:17.474335 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:17.474308    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy" containerID="cri-o://6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5" gracePeriod=30
Jul 09 16:42:17.869867 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:17.869823    2558 patch_prober.go:28] interesting pod/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 16:42:17.870291 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:17.869898    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 16:42:18.534908 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.524720    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"]
Jul 09 16:42:18.534908 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.525018    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" podUID="8ba935b2-c464-4b74-814d-35e3eca4204c" containerName="maas-api" containerID="cri-o://096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707" gracePeriod=30
Jul 09 16:42:18.853052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.853018    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:42:18.870252 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.870215    2558 patch_prober.go:28] interesting pod/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 16:42:18.870633 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.870291    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 16:42:18.943409 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.943370    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba935b2-c464-4b74-814d-35e3eca4204c-openshift-service-ca\") pod \"8ba935b2-c464-4b74-814d-35e3eca4204c\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") "
Jul 09 16:42:18.943616 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.943421    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tdvt2\" (UniqueName: \"kubernetes.io/projected/8ba935b2-c464-4b74-814d-35e3eca4204c-kube-api-access-tdvt2\") pod \"8ba935b2-c464-4b74-814d-35e3eca4204c\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") "
Jul 09 16:42:18.943616 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.943491    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls\") pod \"8ba935b2-c464-4b74-814d-35e3eca4204c\" (UID: \"8ba935b2-c464-4b74-814d-35e3eca4204c\") "
Jul 09 16:42:18.943772 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.943731    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba935b2-c464-4b74-814d-35e3eca4204c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8ba935b2-c464-4b74-814d-35e3eca4204c" (UID: "8ba935b2-c464-4b74-814d-35e3eca4204c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:42:18.943917 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.943900    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ba935b2-c464-4b74-814d-35e3eca4204c-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:18.945775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.945749    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba935b2-c464-4b74-814d-35e3eca4204c-kube-api-access-tdvt2" (OuterVolumeSpecName: "kube-api-access-tdvt2") pod "8ba935b2-c464-4b74-814d-35e3eca4204c" (UID: "8ba935b2-c464-4b74-814d-35e3eca4204c"). InnerVolumeSpecName "kube-api-access-tdvt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:42:18.946180 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:18.946157    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8ba935b2-c464-4b74-814d-35e3eca4204c" (UID: "8ba935b2-c464-4b74-814d-35e3eca4204c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:42:19.044965 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.044928    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ba935b2-c464-4b74-814d-35e3eca4204c-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:19.044965 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.044961    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdvt2\" (UniqueName: \"kubernetes.io/projected/8ba935b2-c464-4b74-814d-35e3eca4204c-kube-api-access-tdvt2\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:19.563440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.563333    2558 generic.go:356] "Generic (PLEG): container finished" podID="8ba935b2-c464-4b74-814d-35e3eca4204c" containerID="096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707" exitCode=0
Jul 09 16:42:19.563440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.563388    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"
Jul 09 16:42:19.563440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.563413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" event={"ID":"8ba935b2-c464-4b74-814d-35e3eca4204c","Type":"ContainerDied","Data":"096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707"}
Jul 09 16:42:19.563714 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.563467    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8" event={"ID":"8ba935b2-c464-4b74-814d-35e3eca4204c","Type":"ContainerDied","Data":"e3bd6d0e02cb3e8e8fd54d244572b7dc638f3535fdc8754e0744af12b6905451"}
Jul 09 16:42:19.563714 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.563489    2558 scope.go:117] "RemoveContainer" containerID="096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707"
Jul 09 16:42:19.573955 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.573931    2558 scope.go:117] "RemoveContainer" containerID="096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707"
Jul 09 16:42:19.574246 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:19.574225    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707\": container with ID starting with 096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707 not found: ID does not exist" containerID="096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707"
Jul 09 16:42:19.574289 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.574256    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707"} err="failed to get container status \"096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707\": rpc error: code = NotFound desc = could not find container \"096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707\": container with ID starting with 096c024028b7a4974449f086571fbf8d8b0e21c128381404e489ec45db308707 not found: ID does not exist"
Jul 09 16:42:19.586003 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.585972    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"]
Jul 09 16:42:19.590028 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.590000    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-0985f7-78445c5f44-slcb8"]
Jul 09 16:42:19.869699 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.869656    2558 patch_prober.go:28] interesting pod/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 16:42:19.869916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:19.869725    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 16:42:20.870050 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:20.870012    2558 patch_prober.go:28] interesting pod/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Jul 09 16:42:20.870477 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:20.870084    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Jul 09 16:42:21.178770 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:21.178673    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba935b2-c464-4b74-814d-35e3eca4204c" path="/var/lib/kubelet/pods/8ba935b2-c464-4b74-814d-35e3eca4204c/volumes"
Jul 09 16:42:21.874496 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:21.874463    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:42:21.874947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:21.874753    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:42:21.875618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:21.875599    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:42:22.746276 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.746186    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:42:22.880018 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.879970    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-credential-socket\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880018 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880026    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-data\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880056    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-token\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880104    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istiod-ca-cert\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880124    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-socket\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880156    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nk5m8\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-kube-api-access-nk5m8\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880184    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-podinfo\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880269    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-certs\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880319    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-envoy\") pod \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\" (UID: \"4681cc3d-30d4-4ee7-b30f-bef3a925b723\") "
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880336    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:22.880558 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880523    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:22.880978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880606    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.880978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880623    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:22.880978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880736    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:42:22.880978 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.880922    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-data" (OuterVolumeSpecName: "istio-data") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:22.882701 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.882674    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:22.882815 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.882712    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-token" (OuterVolumeSpecName: "istio-token") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:42:22.882965 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.882943    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-kube-api-access-nk5m8" (OuterVolumeSpecName: "kube-api-access-nk5m8") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "kube-api-access-nk5m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:42:22.883036 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.882954    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4681cc3d-30d4-4ee7-b30f-bef3a925b723" (UID: "4681cc3d-30d4-4ee7-b30f-bef3a925b723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:42:22.981473 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981423    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981473 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981467    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981473 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981479    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981490    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981498    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981505    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4681cc3d-30d4-4ee7-b30f-bef3a925b723-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981513    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk5m8\" (UniqueName: \"kubernetes.io/projected/4681cc3d-30d4-4ee7-b30f-bef3a925b723-kube-api-access-nk5m8\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:22.981734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:22.981522    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4681cc3d-30d4-4ee7-b30f-bef3a925b723-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:23.588850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.588810    2558 generic.go:356] "Generic (PLEG): container finished" podID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerID="6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5" exitCode=0
Jul 09 16:42:23.589041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.588865    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"
Jul 09 16:42:23.589041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.588909    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" event={"ID":"4681cc3d-30d4-4ee7-b30f-bef3a925b723","Type":"ContainerDied","Data":"6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5"}
Jul 09 16:42:23.589041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.588960    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7" event={"ID":"4681cc3d-30d4-4ee7-b30f-bef3a925b723","Type":"ContainerDied","Data":"94865c36d111193552c833db2598d900305f6772432fa82e935a5f7569441cb0"}
Jul 09 16:42:23.589041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.588983    2558 scope.go:117] "RemoveContainer" containerID="6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5"
Jul 09 16:42:23.598810 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.598608    2558 scope.go:117] "RemoveContainer" containerID="d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"
Jul 09 16:42:23.607918 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.607859    2558 scope.go:117] "RemoveContainer" containerID="6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5"
Jul 09 16:42:23.608291 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:23.608261    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5\": container with ID starting with 6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5 not found: ID does not exist" containerID="6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5"
Jul 09 16:42:23.608347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.608306    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5"} err="failed to get container status \"6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5\": rpc error: code = NotFound desc = could not find container \"6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5\": container with ID starting with 6b0a3c5052a3ba744b4af7d542062b6ec8d733c37fa3a905ebae04e73feaa6c5 not found: ID does not exist"
Jul 09 16:42:23.608347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.608332    2558 scope.go:117] "RemoveContainer" containerID="d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"
Jul 09 16:42:23.608652 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:23.608631    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6\": container with ID starting with d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6 not found: ID does not exist" containerID="d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"
Jul 09 16:42:23.608721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.608659    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6"} err="failed to get container status \"d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6\": rpc error: code = NotFound desc = could not find container \"d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6\": container with ID starting with d3d46956494ecd6d5857c9ddcc231bc0054f5302c2b802a22f33098bd06f10f6 not found: ID does not exist"
Jul 09 16:42:23.610265 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.610240    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"]
Jul 09 16:42:23.613771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:23.613744    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b7d28b-openshift-default-768b747445-rf7g7"]
Jul 09 16:42:24.343577 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:24.343539    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"]
Jul 09 16:42:24.595967 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:24.595848    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy" containerID="cri-o://7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310" gracePeriod=30
Jul 09 16:42:25.180064 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.180028    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" path="/var/lib/kubelet/pods/4681cc3d-30d4-4ee7-b30f-bef3a925b723/volumes"
Jul 09 16:42:25.185751 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.185707    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"]
Jul 09 16:42:25.186112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186090    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821df56f-e1be-43f6-b110-e367aaf64b53" containerName="maas-api"
Jul 09 16:42:25.186112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186109    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="821df56f-e1be-43f6-b110-e367aaf64b53" containerName="maas-api"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186125    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186133    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186159    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba935b2-c464-4b74-814d-35e3eca4204c" containerName="maas-api"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186167    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba935b2-c464-4b74-814d-35e3eca4204c" containerName="maas-api"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186180    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186189    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186280    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="821df56f-e1be-43f6-b110-e367aaf64b53" containerName="maas-api"
Jul 09 16:42:25.186314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186293    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba935b2-c464-4b74-814d-35e3eca4204c" containerName="maas-api"
Jul 09 16:42:25.186747 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186512    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy"
Jul 09 16:42:25.186747 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.186538    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="4681cc3d-30d4-4ee7-b30f-bef3a925b723" containerName="istio-proxy"
Jul 09 16:42:25.191348 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.191318    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.194539 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.194517    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-a36bf9-openshift-default-dockercfg-2bgv7\""
Jul 09 16:42:25.202989 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.202951    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"]
Jul 09 16:42:25.301511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301476    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-certs\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.301511 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301511    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-token\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.301792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301539    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-socket\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.301792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301600    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-envoy\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.301792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301684    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b932c0-f451-4cee-9f12-31a686398ef4-istiod-ca-cert\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.301792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301718    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-822nd\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-kube-api-access-822nd\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.302059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301813    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-credential-socket\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.302059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301869    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-podinfo\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.302059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.301945    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-data\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.402852 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.402806    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-socket\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.402852 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.402849    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-envoy\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403002    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b932c0-f451-4cee-9f12-31a686398ef4-istiod-ca-cert\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403061    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-822nd\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-kube-api-access-822nd\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-credential-socket\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403181    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-podinfo\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403241    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-data\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403288    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-certs\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403313    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-token\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403331    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-socket\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403657    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-data\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b932c0-f451-4cee-9f12-31a686398ef4-istiod-ca-cert\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403738    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-credential-socket\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.403935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.403850    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-certs\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.405290 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.405268    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-envoy\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.405816 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.405779    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-podinfo\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.412678 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.412638    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-token\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.413123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.413099    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-822nd\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-kube-api-access-822nd\") pod \"e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") " pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.502705 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.502611    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:25.654324 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.654295    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"]
Jul 09 16:42:25.656386 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:42:25.656342    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b932c0_f451_4cee_9f12_31a686398ef4.slice/crio-e7e6878df6b0648bc5c7b189ee4da1d4e0b0dfd1b89bcaa47a64abc26f39e462 WatchSource:0}: Error finding container e7e6878df6b0648bc5c7b189ee4da1d4e0b0dfd1b89bcaa47a64abc26f39e462: Status 404 returned error can't find the container with id e7e6878df6b0648bc5c7b189ee4da1d4e0b0dfd1b89bcaa47a64abc26f39e462
Jul 09 16:42:25.658692 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.658655    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:25.658801 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.658726    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:25.658801 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:25.658755    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:26.609740 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:26.609686    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" event={"ID":"a6b932c0-f451-4cee-9f12-31a686398ef4","Type":"ContainerStarted","Data":"743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"}
Jul 09 16:42:26.610242 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:26.609788    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" event={"ID":"a6b932c0-f451-4cee-9f12-31a686398ef4","Type":"ContainerStarted","Data":"e7e6878df6b0648bc5c7b189ee4da1d4e0b0dfd1b89bcaa47a64abc26f39e462"}
Jul 09 16:42:26.638656 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:26.638594    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podStartSLOduration=1.638574675 podStartE2EDuration="1.638574675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:42:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:42:26.636955223 +0000 UTC m=+2292.018772010" watchObservedRunningTime="2026-07-09 16:42:26.638574675 +0000 UTC m=+2292.020391461"
Jul 09 16:42:27.503163 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:27.503122    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:27.508626 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:27.508581    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:27.615935 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:27.615897    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:27.617211 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:27.617182    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:28.601070 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.601024    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"]
Jul 09 16:42:28.604765 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.604732    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.607634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.607605    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-a36bf9-serving-cert\""
Jul 09 16:42:28.607634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.607623    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 16:42:28.607863 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.607709    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-w7wwk\""
Jul 09 16:42:28.609069 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.609042    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 16:42:28.619973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.619942    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"]
Jul 09 16:42:28.733813 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.733781    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcac3a2b-b97e-4398-b031-71e5e102d219-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.734027 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.733937    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznbl\" (UniqueName: \"kubernetes.io/projected/bcac3a2b-b97e-4398-b031-71e5e102d219-kube-api-access-hznbl\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.734079 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.734050    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.835109 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.835066    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.835337 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.835132    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcac3a2b-b97e-4398-b031-71e5e102d219-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.835337 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.835174    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznbl\" (UniqueName: \"kubernetes.io/projected/bcac3a2b-b97e-4398-b031-71e5e102d219-kube-api-access-hznbl\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.835337 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:28.835229    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-serving-cert: secret "maas-api-e2e-shared-a-a36bf9-serving-cert" not found
Jul 09 16:42:28.835337 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:28.835298    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls podName:bcac3a2b-b97e-4398-b031-71e5e102d219 nodeName:}" failed. No retries permitted until 2026-07-09 16:42:29.335281444 +0000 UTC m=+2294.717098211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls") pod "maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" (UID: "bcac3a2b-b97e-4398-b031-71e5e102d219") : secret "maas-api-e2e-shared-a-a36bf9-serving-cert" not found
Jul 09 16:42:28.835941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.835870    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcac3a2b-b97e-4398-b031-71e5e102d219-openshift-service-ca\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:28.844796 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:28.844762    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznbl\" (UniqueName: \"kubernetes.io/projected/bcac3a2b-b97e-4398-b031-71e5e102d219-kube-api-access-hznbl\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:29.340439 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.340395    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:29.343175 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.343137    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls\") pod \"maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:29.516675 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.516633    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:29.668689 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.668662    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"]
Jul 09 16:42:29.671899 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:42:29.671851    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcac3a2b_b97e_4398_b031_71e5e102d219.slice/crio-97a09c36119ed408744e46646744fb6fe4011724a16c5b9eed251e0c68fd12f8 WatchSource:0}: Error finding container 97a09c36119ed408744e46646744fb6fe4011724a16c5b9eed251e0c68fd12f8: Status 404 returned error can't find the container with id 97a09c36119ed408744e46646744fb6fe4011724a16c5b9eed251e0c68fd12f8
Jul 09 16:42:29.848556 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.848517    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:42:29.945347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945232    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-istio-token\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945297    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a832e553-8d47-4d15-9fab-098faccf2b16-istio-podinfo\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945347 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945327    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k5mz7\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-kube-api-access-k5mz7\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945362    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a832e553-8d47-4d15-9fab-098faccf2b16-istiod-ca-cert\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945408    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-credential-socket\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945432    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-data\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945516    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-envoy\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945539    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-certs\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945591    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-socket\") pod \"a832e553-8d47-4d15-9fab-098faccf2b16\" (UID: \"a832e553-8d47-4d15-9fab-098faccf2b16\") "
Jul 09 16:42:29.945947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945701    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:29.945947 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945786    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a832e553-8d47-4d15-9fab-098faccf2b16-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:42:29.946060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945958    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:29.946060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945978    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a832e553-8d47-4d15-9fab-098faccf2b16-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:29.946060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.945999    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:29.946203 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.946167    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:29.946537 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.946512    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-data" (OuterVolumeSpecName: "istio-data") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:29.947930 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.947897    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a832e553-8d47-4d15-9fab-098faccf2b16-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:42:29.948065 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.948010    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-kube-api-access-k5mz7" (OuterVolumeSpecName: "kube-api-access-k5mz7") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "kube-api-access-k5mz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:42:29.948246 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.948221    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-istio-token" (OuterVolumeSpecName: "istio-token") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:42:29.948333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:29.948251    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a832e553-8d47-4d15-9fab-098faccf2b16" (UID: "a832e553-8d47-4d15-9fab-098faccf2b16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:42:30.047015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.046971    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.047015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.047006    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.047015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.047015    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.047015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.047023    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.047015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.047032    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a832e553-8d47-4d15-9fab-098faccf2b16-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.047320 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.047040    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5mz7\" (UniqueName: \"kubernetes.io/projected/a832e553-8d47-4d15-9fab-098faccf2b16-kube-api-access-k5mz7\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.047320 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.047048    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a832e553-8d47-4d15-9fab-098faccf2b16-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:42:30.636996 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.636946    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" event={"ID":"bcac3a2b-b97e-4398-b031-71e5e102d219","Type":"ContainerStarted","Data":"763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9"}
Jul 09 16:42:30.637218 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.637163    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:30.637287 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.637211    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" event={"ID":"bcac3a2b-b97e-4398-b031-71e5e102d219","Type":"ContainerStarted","Data":"97a09c36119ed408744e46646744fb6fe4011724a16c5b9eed251e0c68fd12f8"}
Jul 09 16:42:30.638843 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.638813    2558 generic.go:356] "Generic (PLEG): container finished" podID="a832e553-8d47-4d15-9fab-098faccf2b16" containerID="7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310" exitCode=0
Jul 09 16:42:30.639030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.638852    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" event={"ID":"a832e553-8d47-4d15-9fab-098faccf2b16","Type":"ContainerDied","Data":"7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310"}
Jul 09 16:42:30.639030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.638915    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"
Jul 09 16:42:30.639030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.638932    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b" event={"ID":"a832e553-8d47-4d15-9fab-098faccf2b16","Type":"ContainerDied","Data":"3b0baef53337453b67e3e111bee774185f11791be8285219e7491b5c9080ae18"}
Jul 09 16:42:30.639030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.638963    2558 scope.go:117] "RemoveContainer" containerID="7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310"
Jul 09 16:42:30.649373 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.649346    2558 scope.go:117] "RemoveContainer" containerID="6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"
Jul 09 16:42:30.658540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.658482    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" podStartSLOduration=2.271628007 podStartE2EDuration="2.658465336s" totalImagesPullingTime="386.837329ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:42:30.655730194 +0000 UTC m=+2296.037546982" watchObservedRunningTime="2026-07-09 16:42:30.658465336 +0000 UTC m=+2296.040282121"
Jul 09 16:42:30.660003 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.659983    2558 scope.go:117] "RemoveContainer" containerID="7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310"
Jul 09 16:42:30.660324 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:30.660301    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310\": container with ID starting with 7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310 not found: ID does not exist" containerID="7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310"
Jul 09 16:42:30.660370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.660334    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310"} err="failed to get container status \"7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310\": rpc error: code = NotFound desc = could not find container \"7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310\": container with ID starting with 7b68adb39fbfce31942605449a622c041a3526e951a91ba7b9e14353e45c0310 not found: ID does not exist"
Jul 09 16:42:30.660407 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.660373    2558 scope.go:117] "RemoveContainer" containerID="6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"
Jul 09 16:42:30.660650 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:30.660627    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88\": container with ID starting with 6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88 not found: ID does not exist" containerID="6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"
Jul 09 16:42:30.660725 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.660660    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88"} err="failed to get container status \"6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88\": rpc error: code = NotFound desc = could not find container \"6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88\": container with ID starting with 6eae5d60603d87bdf9ced203c55322fc30f9e95ad1e29406d5a106cefb252d88 not found: ID does not exist"
Jul 09 16:42:30.671221 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.671183    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"]
Jul 09 16:42:30.674986 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:30.674951    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0985f7-openshift-default-7d6f4c7855-44x5b"]
Jul 09 16:42:31.178241 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:31.178198    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" path="/var/lib/kubelet/pods/a832e553-8d47-4d15-9fab-098faccf2b16/volumes"
Jul 09 16:42:33.657434 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.657402    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"]
Jul 09 16:42:33.658072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.657953    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy"
Jul 09 16:42:33.658072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.658008    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy"
Jul 09 16:42:33.658072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.658033    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy"
Jul 09 16:42:33.658072 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.658042    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy"
Jul 09 16:42:33.658377 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.658161    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy"
Jul 09 16:42:33.658377 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.658176    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a832e553-8d47-4d15-9fab-098faccf2b16" containerName="istio-proxy"
Jul 09 16:42:33.663527 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.663496    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.666599 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.666573    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-fb02fd-openshift-default-dockercfg-k9l6x\""
Jul 09 16:42:33.675060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.675031    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"]
Jul 09 16:42:33.783632 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783593    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-envoy\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.783844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783641    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-socket\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.783844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783673    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-credential-socket\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.783844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783694    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-certs\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.783844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783722    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxsd\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-kube-api-access-4bxsd\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.784059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783841    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-token\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.784059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783915    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-data\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.784059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.783944    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-podinfo\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.784059 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.784052    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd244e0-2708-459c-9fdd-42093e7661bd-istiod-ca-cert\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885047 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885004    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-data\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885047 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885044    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-podinfo\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885090    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd244e0-2708-459c-9fdd-42093e7661bd-istiod-ca-cert\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885128    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-envoy\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-socket\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885280    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-credential-socket\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885332 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-certs\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885349    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxsd\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-kube-api-access-4bxsd\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885413    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-token\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885500    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-data\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885595 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885523    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-socket\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885678    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-credential-socket\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.885775 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.885728    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-certs\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.886027 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.886010    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd244e0-2708-459c-9fdd-42093e7661bd-istiod-ca-cert\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.887747 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.887721    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-envoy\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.887967 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.887949    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-podinfo\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.895139 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.895103    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-token\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.895404 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.895372    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxsd\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-kube-api-access-4bxsd\") pod \"e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") " pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:33.975640 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:33.975518    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:34.131714 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.131682    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"]
Jul 09 16:42:34.133079 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:42:34.133036    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd244e0_2708_459c_9fdd_42093e7661bd.slice/crio-bbb988a56b592d3f4deda89c25206a0af6d5757e4cd80fdd8eff10779a891a2e WatchSource:0}: Error finding container bbb988a56b592d3f4deda89c25206a0af6d5757e4cd80fdd8eff10779a891a2e: Status 404 returned error can't find the container with id bbb988a56b592d3f4deda89c25206a0af6d5757e4cd80fdd8eff10779a891a2e
Jul 09 16:42:34.135608 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.135570    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:34.135691 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.135654    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:34.135725 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.135688    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:34.666907 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.666853    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" event={"ID":"ebd244e0-2708-459c-9fdd-42093e7661bd","Type":"ContainerStarted","Data":"38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"}
Jul 09 16:42:34.667350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.666940    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" event={"ID":"ebd244e0-2708-459c-9fdd-42093e7661bd","Type":"ContainerStarted","Data":"bbb988a56b592d3f4deda89c25206a0af6d5757e4cd80fdd8eff10779a891a2e"}
Jul 09 16:42:34.688623 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.688571    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podStartSLOduration=1.688541716 podStartE2EDuration="1.688541716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:42:34.687362492 +0000 UTC m=+2300.069179272" watchObservedRunningTime="2026-07-09 16:42:34.688541716 +0000 UTC m=+2300.070358502"
Jul 09 16:42:34.976553 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:34.976427    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:35.981997 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:35.981963    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:36.652618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.652582    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:42:36.679924 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.679873    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:36.680893 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.680855    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:36.776501 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.776454    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"]
Jul 09 16:42:36.780427 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.780396    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:36.783533 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.783504    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-fb02fd-serving-cert\""
Jul 09 16:42:36.791199 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.791170    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"]
Jul 09 16:42:36.939529 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.939422    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:36.939700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.939532    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:36.939700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:36.939559    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46t7p\" (UniqueName: \"kubernetes.io/projected/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-kube-api-access-46t7p\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.040771 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.040731    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.041301 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:37.040942    2558 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-serving-cert: secret "maas-api-e2e-shared-b-fb02fd-serving-cert" not found
Jul 09 16:42:37.041370 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:42:37.041358    2558 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls podName:83f50dd2-0d3b-4be7-8b30-cf771dd3de4e nodeName:}" failed. No retries permitted until 2026-07-09 16:42:37.541337612 +0000 UTC m=+2302.923154379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls") pod "maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" (UID: "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e") : secret "maas-api-e2e-shared-b-fb02fd-serving-cert" not found
Jul 09 16:42:37.041419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.041293    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.041419 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.041408    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t7p\" (UniqueName: \"kubernetes.io/projected/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-kube-api-access-46t7p\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.041998 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.041980    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.053041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.052964    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t7p\" (UniqueName: \"kubernetes.io/projected/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-kube-api-access-46t7p\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.547056 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.546999    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.549703 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.549675    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls\") pod \"maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.695301 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.695191    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:37.896043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:37.896019    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"]
Jul 09 16:42:37.897872 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:42:37.897840    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f50dd2_0d3b_4be7_8b30_cf771dd3de4e.slice/crio-ed5ee8b5c379a214a6564dac9a516fe50417c3eb21042c5d3befe0b811c896bf WatchSource:0}: Error finding container ed5ee8b5c379a214a6564dac9a516fe50417c3eb21042c5d3befe0b811c896bf: Status 404 returned error can't find the container with id ed5ee8b5c379a214a6564dac9a516fe50417c3eb21042c5d3befe0b811c896bf
Jul 09 16:42:38.691447 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:38.691398    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" event={"ID":"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e","Type":"ContainerStarted","Data":"ed5ee8b5c379a214a6564dac9a516fe50417c3eb21042c5d3befe0b811c896bf"}
Jul 09 16:42:41.716256 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:41.716208    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" event={"ID":"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e","Type":"ContainerStarted","Data":"d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290"}
Jul 09 16:42:41.716814 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:41.716296    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:41.736442 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:41.736377    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" podStartSLOduration=2.721415938 podStartE2EDuration="5.736360254s" totalImagesPullingTime="3.014944316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:42:41.734492987 +0000 UTC m=+2307.116309880" watchObservedRunningTime="2026-07-09 16:42:41.736360254 +0000 UTC m=+2307.118177040"
Jul 09 16:42:42.883312 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:42.882999    2558 patch_prober.go:28] interesting pod/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": read tcp 10.134.0.2:55052->10.134.0.69:15021: read: connection reset by peer" start-of-body=
Jul 09 16:42:42.883312 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:42.883071    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": read tcp 10.134.0.2:55052->10.134.0.69:15021: read: connection reset by peer"
Jul 09 16:42:43.738160 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:43.738122    2558 generic.go:356] "Generic (PLEG): container finished" podID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerID="743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1" exitCode=0
Jul 09 16:42:43.738369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:43.738214    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" event={"ID":"a6b932c0-f451-4cee-9f12-31a686398ef4","Type":"ContainerDied","Data":"743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"}
Jul 09 16:42:43.738710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:43.738696    2558 scope.go:117] "RemoveContainer" containerID="743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"
Jul 09 16:42:43.739839 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:43.739810    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:43.739926 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:43.739904    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:43.739969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:43.739942    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:44.750342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:44.750302    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" event={"ID":"a6b932c0-f451-4cee-9f12-31a686398ef4","Type":"ContainerStarted","Data":"b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5"}
Jul 09 16:42:45.503039 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:45.502997    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:45.503243 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:45.503201    2558 patch_prober.go:28] interesting pod/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 09 16:42:45.503293 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:45.503260    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 09 16:42:46.503861 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:46.503758    2558 patch_prober.go:28] interesting pod/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 09 16:42:46.503861 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:46.503821    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 09 16:42:47.504001 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:47.503960    2558 patch_prober.go:28] interesting pod/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 09 16:42:47.504495 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:47.504048    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 09 16:42:47.736636 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:47.736599    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:42:48.503895 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:48.503850    2558 patch_prober.go:28] interesting pod/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Jul 09 16:42:48.504085 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:48.503943    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Jul 09 16:42:49.507032 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.506998    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:49.507534 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.507391    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:49.508351 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.508334    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:42:49.786874 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.786791    2558 generic.go:356] "Generic (PLEG): container finished" podID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerID="38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d" exitCode=0
Jul 09 16:42:49.787074 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.786867    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" event={"ID":"ebd244e0-2708-459c-9fdd-42093e7661bd","Type":"ContainerDied","Data":"38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"}
Jul 09 16:42:49.787430 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.787413    2558 scope.go:117] "RemoveContainer" containerID="38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"
Jul 09 16:42:49.788514 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.788485    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:49.788593 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.788576    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:49.788646 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:49.788631    2558 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892168Ki","pods":"250"}
Jul 09 16:42:50.801456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:50.801421    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" event={"ID":"ebd244e0-2708-459c-9fdd-42093e7661bd","Type":"ContainerStarted","Data":"e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb"}
Jul 09 16:42:50.975977 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:50.975935    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:50.977230 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:50.977205    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 09 16:42:50.977294 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:50.977261    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 09 16:42:51.680536 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:51.680491    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:51.976299 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:51.976203    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 09 16:42:51.976299 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:51.976277    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 09 16:42:52.976193 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:52.976157    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 09 16:42:52.976370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:52.976226    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 09 16:42:53.976008 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:53.975975    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 09 16:42:53.976200 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:53.976030    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 09 16:42:54.976159 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:54.976119    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 09 16:42:54.976576 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:54.976179    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 09 16:42:55.976295 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:55.976253    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Jul 09 16:42:55.976791 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:55.976325    2558 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Jul 09 16:42:56.980845 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:56.980804    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:56.981824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:56.981804    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:42:57.257985 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.257872    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"]
Jul 09 16:42:57.261755 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.261736    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.264612 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.264582    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a36bf9\"/\"openshift-service-ca.crt\""
Jul 09 16:42:57.264785 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.264759    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-a36bf9\"/\"kube-root-ca.crt\""
Jul 09 16:42:57.264946 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.264930    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a36bf9\"/\"default-dockercfg-977fj\""
Jul 09 16:42:57.266134 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.266111    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a36bf9\"/\"auth-test-model-a36bf9-kserve-self-signed-certs\""
Jul 09 16:42:57.272273 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.272208    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"]
Jul 09 16:42:57.365250 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.365215    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tmp-dir\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.365441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.365288    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-home\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.365441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.365339    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-dshm\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.365441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.365359    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tls-certs\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.365441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.365377    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2zjg\" (UniqueName: \"kubernetes.io/projected/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-kube-api-access-j2zjg\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.365667 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.365527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-model-cache\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466102    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-home\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466155    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-dshm\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466193    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tls-certs\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2zjg\" (UniqueName: \"kubernetes.io/projected/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-kube-api-access-j2zjg\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466285    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-model-cache\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466316    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tmp-dir\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466769 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466715    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tmp-dir\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.466868 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.466847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-model-cache\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.467169 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.467139    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-home\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.469830 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.469773    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-dshm\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.471044 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.470075    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tls-certs\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.476159 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.476133    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2zjg\" (UniqueName: \"kubernetes.io/projected/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-kube-api-access-j2zjg\") pod \"auth-test-model-a36bf9-kserve-6c4fc69767-blfz5\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") " pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.574142 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.574099    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:42:57.714348 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.714321    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"]
Jul 09 16:42:57.715579 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:42:57.715547    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf4ce2b6_09a8_4b6f_b4c2_6074d616c6c4.slice/crio-624dcad87a55d2699b33499293c478cf200828e06a973f29d0ac5b3e0991d2eb WatchSource:0}: Error finding container 624dcad87a55d2699b33499293c478cf200828e06a973f29d0ac5b3e0991d2eb: Status 404 returned error can't find the container with id 624dcad87a55d2699b33499293c478cf200828e06a973f29d0ac5b3e0991d2eb
Jul 09 16:42:57.855925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.855805    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5" event={"ID":"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4","Type":"ContainerStarted","Data":"f97adf6895e62e396347049e126c993559049f0a8a574c39271ecf6eeea37020"}
Jul 09 16:42:57.855925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.855856    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5" event={"ID":"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4","Type":"ContainerStarted","Data":"624dcad87a55d2699b33499293c478cf200828e06a973f29d0ac5b3e0991d2eb"}
Jul 09 16:42:57.873631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:42:57.873578    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5" podStartSLOduration=0.873561165 podStartE2EDuration="873.561165ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:42:57.872097903 +0000 UTC m=+2323.253914692" watchObservedRunningTime="2026-07-09 16:42:57.873561165 +0000 UTC m=+2323.255377952"
Jul 09 16:43:07.575313 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:07.575188    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:43:07.575313 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:07.575252    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:43:07.588458 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:07.588422    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:43:07.922510 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:07.922408    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:43:16.170548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.170501    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"]
Jul 09 16:43:16.175628 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.175591    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.178897 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.178855    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"kube-root-ca.crt\""
Jul 09 16:43:16.180433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.180410    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"default-dockercfg-spfrn\""
Jul 09 16:43:16.180610 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.180410    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"auth-test-model-fb02fd-kserve-self-signed-certs\""
Jul 09 16:43:16.180819 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.180797    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"openshift-service-ca.crt\""
Jul 09 16:43:16.184107 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.184083    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"]
Jul 09 16:43:16.248824 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.248783    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tmp-dir\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.249052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.248834    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2kq4\" (UniqueName: \"kubernetes.io/projected/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-kube-api-access-x2kq4\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.249052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.248913    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-home\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.249052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.248959    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-dshm\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.249052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.248981    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-model-cache\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.249190 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.249090    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tls-certs\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350273 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350224    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tls-certs\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350464 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350309    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tmp-dir\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350464 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350351    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2kq4\" (UniqueName: \"kubernetes.io/projected/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-kube-api-access-x2kq4\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350500    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-home\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350599 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350581    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-dshm\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350643 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350614    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-model-cache\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.350844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350819    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tmp-dir\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.351016 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.350847    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-home\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.351255 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.351039    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-model-cache\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.352873 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.352842    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-dshm\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.353026 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.353000    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tls-certs\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.359233 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.359207    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2kq4\" (UniqueName: \"kubernetes.io/projected/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-kube-api-access-x2kq4\") pod \"auth-test-model-fb02fd-kserve-5d58788c6c-565sq\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") " pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.487711 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.487609    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:16.643355 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.643325    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"]
Jul 09 16:43:16.645430 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:43:16.645394    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9ae74b1_ad37_4356_9e35_5b2cfca1ed79.slice/crio-c64ed2aaf45fd9b89e989867f892058733178fafed42bb3db02fe672cfa27e61 WatchSource:0}: Error finding container c64ed2aaf45fd9b89e989867f892058733178fafed42bb3db02fe672cfa27e61: Status 404 returned error can't find the container with id c64ed2aaf45fd9b89e989867f892058733178fafed42bb3db02fe672cfa27e61
Jul 09 16:43:16.966594 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.966552    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq" event={"ID":"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79","Type":"ContainerStarted","Data":"1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a"}
Jul 09 16:43:16.966594 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.966601    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq" event={"ID":"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79","Type":"ContainerStarted","Data":"c64ed2aaf45fd9b89e989867f892058733178fafed42bb3db02fe672cfa27e61"}
Jul 09 16:43:16.984247 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:16.984169    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq" podStartSLOduration=0.984150591 podStartE2EDuration="984.150591ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:43:16.983075754 +0000 UTC m=+2342.364892543" watchObservedRunningTime="2026-07-09 16:43:16.984150591 +0000 UTC m=+2342.365967379"
Jul 09 16:43:26.488633 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:26.488577    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:26.488633 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:26.488637    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:26.501873 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:26.501839    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:43:27.039669 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:43:27.039635    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:44:15.234583 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:44:15.234558    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:44:15.237190 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:44:15.237171    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:45:20.462048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:20.462011    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"]
Jul 09 16:45:21.346226 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:21.346195    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"]
Jul 09 16:45:30.858626 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.858595    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"]
Jul 09 16:45:30.862108 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.862091    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:30.865353 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.865331    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a36bf9\"/\"sub-test-model-a36bf9-kserve-self-signed-certs\""
Jul 09 16:45:30.872490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.872467    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"]
Jul 09 16:45:30.903449 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.903425    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288e4e86-234e-447f-af41-0e30a193ef22-tls-certs\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:30.903543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.903458    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-tmp-dir\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:30.903543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.903490    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4pv\" (UniqueName: \"kubernetes.io/projected/288e4e86-234e-447f-af41-0e30a193ef22-kube-api-access-dk4pv\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:30.903543 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.903533    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-model-cache\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:30.903661 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.903614    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-home\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:30.903661 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:30.903631    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-dshm\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004235 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004204    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288e4e86-234e-447f-af41-0e30a193ef22-tls-certs\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004392 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004246    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-tmp-dir\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004392 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004372    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4pv\" (UniqueName: \"kubernetes.io/projected/288e4e86-234e-447f-af41-0e30a193ef22-kube-api-access-dk4pv\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004468 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004405    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-model-cache\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004478    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-home\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004541 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004499    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-dshm\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004635    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-tmp-dir\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004770 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004753    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-model-cache\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.004826 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.004785    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-home\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.006741 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.006711    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-dshm\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.007025 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.007005    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288e4e86-234e-447f-af41-0e30a193ef22-tls-certs\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.012941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.012873    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4pv\" (UniqueName: \"kubernetes.io/projected/288e4e86-234e-447f-af41-0e30a193ef22-kube-api-access-dk4pv\") pod \"sub-test-model-a36bf9-kserve-5776654cb5-gcxxs\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") " pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.171734 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.171638    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:31.312143 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.312115    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"]
Jul 09 16:45:31.313083 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:45:31.313059    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod288e4e86_234e_447f_af41_0e30a193ef22.slice/crio-a53a4ef4dd6b1c3356ea4137a27fcefe43d012edd23f203147147205c52d7a3e WatchSource:0}: Error finding container a53a4ef4dd6b1c3356ea4137a27fcefe43d012edd23f203147147205c52d7a3e: Status 404 returned error can't find the container with id a53a4ef4dd6b1c3356ea4137a27fcefe43d012edd23f203147147205c52d7a3e
Jul 09 16:45:31.707691 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.707650    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs" event={"ID":"288e4e86-234e-447f-af41-0e30a193ef22","Type":"ContainerStarted","Data":"32bd001254aeba038ec96c64028ad424f818a73b33b019e8a6cf96816b910236"}
Jul 09 16:45:31.707923 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.707704    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs" event={"ID":"288e4e86-234e-447f-af41-0e30a193ef22","Type":"ContainerStarted","Data":"a53a4ef4dd6b1c3356ea4137a27fcefe43d012edd23f203147147205c52d7a3e"}
Jul 09 16:45:31.726187 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:31.726143    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs" podStartSLOduration=1.726127913 podStartE2EDuration="1.726127913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:45:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:45:31.725086698 +0000 UTC m=+2477.106903486" watchObservedRunningTime="2026-07-09 16:45:31.726127913 +0000 UTC m=+2477.107944700"
Jul 09 16:45:35.482052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.482008    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5" podUID="df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" containerName="main" containerID="cri-o://f97adf6895e62e396347049e126c993559049f0a8a574c39271ecf6eeea37020" gracePeriod=45
Jul 09 16:45:35.732483 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.732423    2558 generic.go:356] "Generic (PLEG): container finished" podID="df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" containerID="f97adf6895e62e396347049e126c993559049f0a8a574c39271ecf6eeea37020" exitCode=0
Jul 09 16:45:35.732617 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.732496    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5" event={"ID":"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4","Type":"ContainerDied","Data":"f97adf6895e62e396347049e126c993559049f0a8a574c39271ecf6eeea37020"}
Jul 09 16:45:35.732617 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.732539    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5" event={"ID":"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4","Type":"ContainerDied","Data":"624dcad87a55d2699b33499293c478cf200828e06a973f29d0ac5b3e0991d2eb"}
Jul 09 16:45:35.732617 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.732550    2558 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="624dcad87a55d2699b33499293c478cf200828e06a973f29d0ac5b3e0991d2eb"
Jul 09 16:45:35.740137 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.740118    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:45:35.851713 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.851682    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tls-certs\") pod \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") "
Jul 09 16:45:35.851916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.851725    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-home\") pod \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") "
Jul 09 16:45:35.851916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.851758    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tmp-dir\") pod \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") "
Jul 09 16:45:35.851916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.851818    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-model-cache\") pod \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") "
Jul 09 16:45:35.851916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.851848    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2zjg\" (UniqueName: \"kubernetes.io/projected/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-kube-api-access-j2zjg\") pod \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") "
Jul 09 16:45:35.851916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.851908    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-dshm\") pod \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\" (UID: \"df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4\") "
Jul 09 16:45:35.852171 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.852028    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-home" (OuterVolumeSpecName: "home") pod "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" (UID: "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:35.852171 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.852075    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" (UID: "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:35.852363 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.852343    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:35.852457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.852369    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:35.852457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.852386    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-model-cache" (OuterVolumeSpecName: "model-cache") pod "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" (UID: "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:35.854180 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.854154    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" (UID: "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:45:35.854278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.854173    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-dshm" (OuterVolumeSpecName: "dshm") pod "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" (UID: "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:35.854278 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.854188    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-kube-api-access-j2zjg" (OuterVolumeSpecName: "kube-api-access-j2zjg") pod "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" (UID: "df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4"). InnerVolumeSpecName "kube-api-access-j2zjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:45:35.953155 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.953125    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:35.953155 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.953151    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2zjg\" (UniqueName: \"kubernetes.io/projected/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-kube-api-access-j2zjg\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:35.953328 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.953162    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:35.953328 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:35.953171    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:36.366112 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.366056    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq" podUID="d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" containerName="main" containerID="cri-o://1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a" gracePeriod=45
Jul 09 16:45:36.626862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.626801    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:45:36.739806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.739774    2558 generic.go:356] "Generic (PLEG): container finished" podID="d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" containerID="1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a" exitCode=0
Jul 09 16:45:36.739983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.739820    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"
Jul 09 16:45:36.739983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.739862    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq" event={"ID":"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79","Type":"ContainerDied","Data":"1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a"}
Jul 09 16:45:36.739983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.739928    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq" event={"ID":"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79","Type":"ContainerDied","Data":"c64ed2aaf45fd9b89e989867f892058733178fafed42bb3db02fe672cfa27e61"}
Jul 09 16:45:36.739983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.739949    2558 scope.go:117] "RemoveContainer" containerID="1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a"
Jul 09 16:45:36.739983 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.739957    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"
Jul 09 16:45:36.752217 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.752198    2558 scope.go:117] "RemoveContainer" containerID="1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a"
Jul 09 16:45:36.752584 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:45:36.752542    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a\": container with ID starting with 1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a not found: ID does not exist" containerID="1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a"
Jul 09 16:45:36.752662 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.752595    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a"} err="failed to get container status \"1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a\": rpc error: code = NotFound desc = could not find container \"1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a\": container with ID starting with 1b823e7ce14b64060d4fd5ac5d9a2e17e2e66ec3bfc20ad3ead1162c74f7a99a not found: ID does not exist"
Jul 09 16:45:36.760028 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760006    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2kq4\" (UniqueName: \"kubernetes.io/projected/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-kube-api-access-x2kq4\") pod \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") "
Jul 09 16:45:36.760122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760050    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tls-certs\") pod \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") "
Jul 09 16:45:36.760122 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760106    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tmp-dir\") pod \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") "
Jul 09 16:45:36.760245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760172    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-dshm\") pod \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") "
Jul 09 16:45:36.760245 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760233    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-home\") pod \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") "
Jul 09 16:45:36.760350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760284    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-model-cache\") pod \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\" (UID: \"d9ae74b1-ad37-4356-9e35-5b2cfca1ed79\") "
Jul 09 16:45:36.760436 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760409    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" (UID: "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:36.760527 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760503    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-home" (OuterVolumeSpecName: "home") pod "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" (UID: "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:36.760625 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-model-cache" (OuterVolumeSpecName: "model-cache") pod "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" (UID: "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:36.760838 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760810    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:36.760941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.760842    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:36.762366 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.762342    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-kube-api-access-x2kq4" (OuterVolumeSpecName: "kube-api-access-x2kq4") pod "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" (UID: "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79"). InnerVolumeSpecName "kube-api-access-x2kq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:45:36.762446 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.762381    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-dshm" (OuterVolumeSpecName: "dshm") pod "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" (UID: "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:45:36.762446 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.762388    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" (UID: "d9ae74b1-ad37-4356-9e35-5b2cfca1ed79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:45:36.789195 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.789167    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"]
Jul 09 16:45:36.796187 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.796167    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/auth-test-model-a36bf9-kserve-6c4fc69767-blfz5"]
Jul 09 16:45:36.861792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.861764    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:36.861792 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.861788    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:36.861969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.861802    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2kq4\" (UniqueName: \"kubernetes.io/projected/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-kube-api-access-x2kq4\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:36.861969 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:36.861814    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:45:37.062654 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:37.062624    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"]
Jul 09 16:45:37.066527 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:37.066504    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/auth-test-model-fb02fd-kserve-5d58788c6c-565sq"]
Jul 09 16:45:37.178679 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:37.178649    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" path="/var/lib/kubelet/pods/d9ae74b1-ad37-4356-9e35-5b2cfca1ed79/volumes"
Jul 09 16:45:37.179128 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:37.179113    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" path="/var/lib/kubelet/pods/df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4/volumes"
Jul 09 16:45:41.179527 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:41.179500    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:41.179912 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:41.179542    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:41.185369 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:41.185347    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:41.777960 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:41.777933    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:45:50.266694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.266648    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"]
Jul 09 16:45:50.267220 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.267204    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" containerName="main"
Jul 09 16:45:50.267270 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.267224    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" containerName="main"
Jul 09 16:45:50.267270 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.267258    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" containerName="main"
Jul 09 16:45:50.267270 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.267267    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" containerName="main"
Jul 09 16:45:50.267370 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.267358    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="df4ce2b6-09a8-4b6f-b4c2-6074d616c6c4" containerName="main"
Jul 09 16:45:50.267404 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.267376    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9ae74b1-ad37-4356-9e35-5b2cfca1ed79" containerName="main"
Jul 09 16:45:50.270782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.270766    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.275415 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.275337    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"openshift-service-ca.crt\""
Jul 09 16:45:50.275660 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.275631    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"sub-test-model-fb02fd-kserve-self-signed-certs\""
Jul 09 16:45:50.275756 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.275678    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"kube-root-ca.crt\""
Jul 09 16:45:50.275963 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.275942    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"default-dockercfg-spfrn\""
Jul 09 16:45:50.279540 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.279519    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"]
Jul 09 16:45:50.380634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.380602    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-model-cache\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.380634 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.380640    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-dshm\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.380844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.380682    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n4nt\" (UniqueName: \"kubernetes.io/projected/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-kube-api-access-5n4nt\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.380844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.380742    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tls-certs\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.380844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.380775    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tmp-dir\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.380844 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.380809    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-home\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482088 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482051    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n4nt\" (UniqueName: \"kubernetes.io/projected/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-kube-api-access-5n4nt\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482293 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482097    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tls-certs\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482293 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482116    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tmp-dir\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482293 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482141    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-home\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482293 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482199    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-model-cache\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482293 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482217    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-dshm\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482572 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482520    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tmp-dir\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482573    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-home\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.482674 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.482647    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-model-cache\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.484519 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.484490    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-dshm\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.484713 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.484698    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tls-certs\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.491115 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.491094    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n4nt\" (UniqueName: \"kubernetes.io/projected/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-kube-api-access-5n4nt\") pod \"sub-test-model-fb02fd-kserve-855679bc75-8x2tn\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") " pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.582615 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.582589    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:45:50.715144 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:45:50.715099    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded288b7d_aa2c_44b4_bc9c_dd3d7eb97996.slice/crio-f206589555a2c0d9ed187695dbeb12d4d4c45ba078e0427c095172a847e70e7d WatchSource:0}: Error finding container f206589555a2c0d9ed187695dbeb12d4d4c45ba078e0427c095172a847e70e7d: Status 404 returned error can't find the container with id f206589555a2c0d9ed187695dbeb12d4d4c45ba078e0427c095172a847e70e7d
Jul 09 16:45:50.715144 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.715114    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"]
Jul 09 16:45:50.823479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.823416    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn" event={"ID":"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996","Type":"ContainerStarted","Data":"64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924"}
Jul 09 16:45:50.823685 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.823497    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn" event={"ID":"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996","Type":"ContainerStarted","Data":"f206589555a2c0d9ed187695dbeb12d4d4c45ba078e0427c095172a847e70e7d"}
Jul 09 16:45:50.842023 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:45:50.841935    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn" podStartSLOduration=0.841921121 podStartE2EDuration="841.921121ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:45:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:45:50.841275883 +0000 UTC m=+2496.223092668" watchObservedRunningTime="2026-07-09 16:45:50.841921121 +0000 UTC m=+2496.223737908"
Jul 09 16:46:00.583195 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:00.583157    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:46:00.583587 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:00.583210    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:46:00.595580 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:00.595557    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:46:00.887522 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:00.887445    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:46:26.890737 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:26.890709    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"]
Jul 09 16:46:27.836729 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:27.836698    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"]
Jul 09 16:46:35.364356 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.364321    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"]
Jul 09 16:46:35.367941 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.367921    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.370957 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.370939    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-a36bf9\"/\"rate-test-model-a36bf9-kserve-self-signed-certs\""
Jul 09 16:46:35.376063 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.376037    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"]
Jul 09 16:46:35.463928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.463892    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-dshm\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.463928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.463924    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd6sg\" (UniqueName: \"kubernetes.io/projected/f2d4e516-45b8-4f83-9610-589825ea5d90-kube-api-access-fd6sg\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.464121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.463948    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-model-cache\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.464121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.463964    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-tmp-dir\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.464121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.464055    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-home\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.464121 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.464101    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d4e516-45b8-4f83-9610-589825ea5d90-tls-certs\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.565557 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.565522    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-dshm\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.565557 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.565562    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd6sg\" (UniqueName: \"kubernetes.io/projected/f2d4e516-45b8-4f83-9610-589825ea5d90-kube-api-access-fd6sg\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.565838 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.565585    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-model-cache\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.565838 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.565605    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-tmp-dir\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.565838 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.565648    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-home\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.565838 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.565699    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d4e516-45b8-4f83-9610-589825ea5d90-tls-certs\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.566060 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.566038    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-model-cache\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.566561 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.566412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-tmp-dir\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.566561 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.566485    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-home\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.573938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.573914    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-dshm\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.574172 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.574152    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d4e516-45b8-4f83-9610-589825ea5d90-tls-certs\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.575975 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.575955    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd6sg\" (UniqueName: \"kubernetes.io/projected/f2d4e516-45b8-4f83-9610-589825ea5d90-kube-api-access-fd6sg\") pod \"rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") " pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.679104 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.679042    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:35.815594 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:35.815572    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"]
Jul 09 16:46:35.817565 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:46:35.817531    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2d4e516_45b8_4f83_9610_589825ea5d90.slice/crio-787a397e7f565cec1b1d89c78e788db62d406119d1f24567e36219b1606658a7 WatchSource:0}: Error finding container 787a397e7f565cec1b1d89c78e788db62d406119d1f24567e36219b1606658a7: Status 404 returned error can't find the container with id 787a397e7f565cec1b1d89c78e788db62d406119d1f24567e36219b1606658a7
Jul 09 16:46:36.063720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:36.063630    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn" event={"ID":"f2d4e516-45b8-4f83-9610-589825ea5d90","Type":"ContainerStarted","Data":"8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718"}
Jul 09 16:46:36.063720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:36.063675    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn" event={"ID":"f2d4e516-45b8-4f83-9610-589825ea5d90","Type":"ContainerStarted","Data":"787a397e7f565cec1b1d89c78e788db62d406119d1f24567e36219b1606658a7"}
Jul 09 16:46:36.081346 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:36.081302    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn" podStartSLOduration=1.081283612 podStartE2EDuration="1.081283612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:46:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:46:36.080257882 +0000 UTC m=+2541.462074669" watchObservedRunningTime="2026-07-09 16:46:36.081283612 +0000 UTC m=+2541.463100400"
Jul 09 16:46:41.911756 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:41.911720    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs" podUID="288e4e86-234e-447f-af41-0e30a193ef22" containerName="main" containerID="cri-o://32bd001254aeba038ec96c64028ad424f818a73b33b019e8a6cf96816b910236" gracePeriod=45
Jul 09 16:46:42.110479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.110286    2558 generic.go:356] "Generic (PLEG): container finished" podID="288e4e86-234e-447f-af41-0e30a193ef22" containerID="32bd001254aeba038ec96c64028ad424f818a73b33b019e8a6cf96816b910236" exitCode=0
Jul 09 16:46:42.110479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.110431    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs" event={"ID":"288e4e86-234e-447f-af41-0e30a193ef22","Type":"ContainerDied","Data":"32bd001254aeba038ec96c64028ad424f818a73b33b019e8a6cf96816b910236"}
Jul 09 16:46:42.225571 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.225546    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:46:42.329094 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329066    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288e4e86-234e-447f-af41-0e30a193ef22-tls-certs\") pod \"288e4e86-234e-447f-af41-0e30a193ef22\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") "
Jul 09 16:46:42.329234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329115    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-tmp-dir\") pod \"288e4e86-234e-447f-af41-0e30a193ef22\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") "
Jul 09 16:46:42.329234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329175    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dk4pv\" (UniqueName: \"kubernetes.io/projected/288e4e86-234e-447f-af41-0e30a193ef22-kube-api-access-dk4pv\") pod \"288e4e86-234e-447f-af41-0e30a193ef22\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") "
Jul 09 16:46:42.329234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329212    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-home\") pod \"288e4e86-234e-447f-af41-0e30a193ef22\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") "
Jul 09 16:46:42.329349 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329257    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-model-cache\") pod \"288e4e86-234e-447f-af41-0e30a193ef22\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") "
Jul 09 16:46:42.329349 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329279    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-dshm\") pod \"288e4e86-234e-447f-af41-0e30a193ef22\" (UID: \"288e4e86-234e-447f-af41-0e30a193ef22\") "
Jul 09 16:46:42.329474 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329425    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "288e4e86-234e-447f-af41-0e30a193ef22" (UID: "288e4e86-234e-447f-af41-0e30a193ef22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:42.329624 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329562    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-model-cache" (OuterVolumeSpecName: "model-cache") pod "288e4e86-234e-447f-af41-0e30a193ef22" (UID: "288e4e86-234e-447f-af41-0e30a193ef22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:42.329708 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.329598    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-home" (OuterVolumeSpecName: "home") pod "288e4e86-234e-447f-af41-0e30a193ef22" (UID: "288e4e86-234e-447f-af41-0e30a193ef22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:42.331378 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.331349    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/288e4e86-234e-447f-af41-0e30a193ef22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "288e4e86-234e-447f-af41-0e30a193ef22" (UID: "288e4e86-234e-447f-af41-0e30a193ef22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:46:42.331462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.331391    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-dshm" (OuterVolumeSpecName: "dshm") pod "288e4e86-234e-447f-af41-0e30a193ef22" (UID: "288e4e86-234e-447f-af41-0e30a193ef22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:42.331462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.331422    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/288e4e86-234e-447f-af41-0e30a193ef22-kube-api-access-dk4pv" (OuterVolumeSpecName: "kube-api-access-dk4pv") pod "288e4e86-234e-447f-af41-0e30a193ef22" (UID: "288e4e86-234e-447f-af41-0e30a193ef22"). InnerVolumeSpecName "kube-api-access-dk4pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:46:42.430409 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.430381    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/288e4e86-234e-447f-af41-0e30a193ef22-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:42.430409 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.430406    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:42.430550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.430415    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk4pv\" (UniqueName: \"kubernetes.io/projected/288e4e86-234e-447f-af41-0e30a193ef22-kube-api-access-dk4pv\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:42.430550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.430423    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:42.430550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.430432    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:42.430550 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.430439    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/288e4e86-234e-447f-af41-0e30a193ef22-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:42.857988 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:42.857924    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn" podUID="ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" containerName="main" containerID="cri-o://64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924" gracePeriod=45
Jul 09 16:46:43.112093 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.112015    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:46:43.116993 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.116970    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"
Jul 09 16:46:43.117127 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.117007    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs" event={"ID":"288e4e86-234e-447f-af41-0e30a193ef22","Type":"ContainerDied","Data":"a53a4ef4dd6b1c3356ea4137a27fcefe43d012edd23f203147147205c52d7a3e"}
Jul 09 16:46:43.117127 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.117077    2558 scope.go:117] "RemoveContainer" containerID="32bd001254aeba038ec96c64028ad424f818a73b33b019e8a6cf96816b910236"
Jul 09 16:46:43.118352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.118331    2558 generic.go:356] "Generic (PLEG): container finished" podID="ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" containerID="64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924" exitCode=0
Jul 09 16:46:43.118462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.118374    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"
Jul 09 16:46:43.118462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.118381    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn" event={"ID":"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996","Type":"ContainerDied","Data":"64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924"}
Jul 09 16:46:43.118462 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.118413    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn" event={"ID":"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996","Type":"ContainerDied","Data":"f206589555a2c0d9ed187695dbeb12d4d4c45ba078e0427c095172a847e70e7d"}
Jul 09 16:46:43.129357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.129339    2558 scope.go:117] "RemoveContainer" containerID="64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924"
Jul 09 16:46:43.140517 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.140495    2558 scope.go:117] "RemoveContainer" containerID="64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924"
Jul 09 16:46:43.140807 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:46:43.140787    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924\": container with ID starting with 64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924 not found: ID does not exist" containerID="64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924"
Jul 09 16:46:43.140916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.140815    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924"} err="failed to get container status \"64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924\": rpc error: code = NotFound desc = could not find container \"64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924\": container with ID starting with 64cd4b3816b4fafef4ca8f46760bc5cd755f79bbe978672fcfd18330d00f8924 not found: ID does not exist"
Jul 09 16:46:43.144003 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.143980    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"]
Jul 09 16:46:43.147761 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.147740    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/sub-test-model-a36bf9-kserve-5776654cb5-gcxxs"]
Jul 09 16:46:43.178325 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.178301    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="288e4e86-234e-447f-af41-0e30a193ef22" path="/var/lib/kubelet/pods/288e4e86-234e-447f-af41-0e30a193ef22/volumes"
Jul 09 16:46:43.237815 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.237791    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-model-cache\") pod \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") "
Jul 09 16:46:43.237937 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.237856    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tls-certs\") pod \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") "
Jul 09 16:46:43.237937 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.237902    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tmp-dir\") pod \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") "
Jul 09 16:46:43.238041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.237936    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n4nt\" (UniqueName: \"kubernetes.io/projected/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-kube-api-access-5n4nt\") pod \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") "
Jul 09 16:46:43.238041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.237965    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-home\") pod \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") "
Jul 09 16:46:43.238041 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238004    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-dshm\") pod \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\" (UID: \"ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996\") "
Jul 09 16:46:43.238198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238085    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" (UID: "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:43.238198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238131    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" (UID: "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:43.238308 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238239    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-home" (OuterVolumeSpecName: "home") pod "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" (UID: "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:43.238356 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238310    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:43.238356 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238330    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:43.238356 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.238347    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:43.240202 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.240182    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-kube-api-access-5n4nt" (OuterVolumeSpecName: "kube-api-access-5n4nt") pod "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" (UID: "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996"). InnerVolumeSpecName "kube-api-access-5n4nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:46:43.240304 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.240199    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" (UID: "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:46:43.240304 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.240276    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-dshm" (OuterVolumeSpecName: "dshm") pod "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" (UID: "ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:46:43.339685 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.339667    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:43.339685 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.339687    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n4nt\" (UniqueName: \"kubernetes.io/projected/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-kube-api-access-5n4nt\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:43.339806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.339697    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:46:43.440855 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.440832    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"]
Jul 09 16:46:43.443234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:43.443214    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/sub-test-model-fb02fd-kserve-855679bc75-8x2tn"]
Jul 09 16:46:45.178111 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:45.178080    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" path="/var/lib/kubelet/pods/ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996/volumes"
Jul 09 16:46:45.680083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:45.680057    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:45.680242 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:45.680091    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:45.692102 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:45.692078    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:46:46.148389 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:46:46.148362    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:47:08.068429 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068391    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"]
Jul 09 16:47:08.068973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068758    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="288e4e86-234e-447f-af41-0e30a193ef22" containerName="main"
Jul 09 16:47:08.068973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068769    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="288e4e86-234e-447f-af41-0e30a193ef22" containerName="main"
Jul 09 16:47:08.068973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068790    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" containerName="main"
Jul 09 16:47:08.068973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068796    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" containerName="main"
Jul 09 16:47:08.068973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068852    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed288b7d-aa2c-44b4-bc9c-dd3d7eb97996" containerName="main"
Jul 09 16:47:08.068973 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.068862    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="288e4e86-234e-447f-af41-0e30a193ef22" containerName="main"
Jul 09 16:47:08.072015 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.071999    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.076227 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.076200    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"kube-root-ca.crt\""
Jul 09 16:47:08.076357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.076251    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"openshift-service-ca.crt\""
Jul 09 16:47:08.076357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.076295    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"default-dockercfg-spfrn\""
Jul 09 16:47:08.076357 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.076298    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-fb02fd\"/\"rate-test-model-fb02fd-kserve-self-signed-certs\""
Jul 09 16:47:08.083283 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.083262    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"]
Jul 09 16:47:08.157244 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.157209    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-home\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.157416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.157258    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tmp-dir\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.157416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.157283    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tls-certs\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.157416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.157355    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cxs\" (UniqueName: \"kubernetes.io/projected/8a615bc0-2be3-4ab6-92d6-1c60953a7792-kube-api-access-69cxs\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.157416 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.157407    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-model-cache\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.157700 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.157441    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-dshm\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.258627 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.258591    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cxs\" (UniqueName: \"kubernetes.io/projected/8a615bc0-2be3-4ab6-92d6-1c60953a7792-kube-api-access-69cxs\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.258832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.258675    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-model-cache\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.258832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.258722    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-dshm\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.258832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.258785    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-home\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.258832 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.258824    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tmp-dir\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.259110 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.258859    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tls-certs\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.259197 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.259174    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-model-cache\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.259261 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.259204    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-home\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.259387 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.259369    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tmp-dir\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.261227 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.261207    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-dshm\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.261946 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.261928    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tls-certs\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.266570 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.266548    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cxs\" (UniqueName: \"kubernetes.io/projected/8a615bc0-2be3-4ab6-92d6-1c60953a7792-kube-api-access-69cxs\") pod \"rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") " pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.381793 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.381706    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:08.522184 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:08.522152    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"]
Jul 09 16:47:08.523419 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:47:08.523393    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a615bc0_2be3_4ab6_92d6_1c60953a7792.slice/crio-67ac2cdb4ecfd9b25c7fcedb119392e34cb1630894ce25ee5cced344f9efbf33 WatchSource:0}: Error finding container 67ac2cdb4ecfd9b25c7fcedb119392e34cb1630894ce25ee5cced344f9efbf33: Status 404 returned error can't find the container with id 67ac2cdb4ecfd9b25c7fcedb119392e34cb1630894ce25ee5cced344f9efbf33
Jul 09 16:47:09.269573 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:09.269528    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx" event={"ID":"8a615bc0-2be3-4ab6-92d6-1c60953a7792","Type":"ContainerStarted","Data":"7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc"}
Jul 09 16:47:09.269992 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:09.269586    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx" event={"ID":"8a615bc0-2be3-4ab6-92d6-1c60953a7792","Type":"ContainerStarted","Data":"67ac2cdb4ecfd9b25c7fcedb119392e34cb1630894ce25ee5cced344f9efbf33"}
Jul 09 16:47:09.287413 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:09.287365    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx" podStartSLOduration=1.287349697 podStartE2EDuration="1.287349697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:47:09.286266597 +0000 UTC m=+2574.668083385" watchObservedRunningTime="2026-07-09 16:47:09.287349697 +0000 UTC m=+2574.669166485"
Jul 09 16:47:18.381846 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:18.381808    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:18.382282 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:18.381862    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:18.394697 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:18.394671    2558 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:19.339972 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:19.339943    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:47:49.237239 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:49.237194    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"]
Jul 09 16:47:50.152214 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:47:50.152178    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"]
Jul 09 16:48:04.258471 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.258402    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn" podUID="f2d4e516-45b8-4f83-9610-589825ea5d90" containerName="main" containerID="cri-o://8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718" gracePeriod=45
Jul 09 16:48:04.511767 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.511684    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:48:04.586254 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.586217    2558 generic.go:356] "Generic (PLEG): container finished" podID="f2d4e516-45b8-4f83-9610-589825ea5d90" containerID="8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718" exitCode=0
Jul 09 16:48:04.586433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.586261    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn" event={"ID":"f2d4e516-45b8-4f83-9610-589825ea5d90","Type":"ContainerDied","Data":"8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718"}
Jul 09 16:48:04.586433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.586281    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"
Jul 09 16:48:04.586433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.586293    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn" event={"ID":"f2d4e516-45b8-4f83-9610-589825ea5d90","Type":"ContainerDied","Data":"787a397e7f565cec1b1d89c78e788db62d406119d1f24567e36219b1606658a7"}
Jul 09 16:48:04.586433 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.586309    2558 scope.go:117] "RemoveContainer" containerID="8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718"
Jul 09 16:48:04.594141 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594119    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-tmp-dir\") pod \"f2d4e516-45b8-4f83-9610-589825ea5d90\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") "
Jul 09 16:48:04.594229 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594166    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d4e516-45b8-4f83-9610-589825ea5d90-tls-certs\") pod \"f2d4e516-45b8-4f83-9610-589825ea5d90\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") "
Jul 09 16:48:04.594229 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594200    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-home\") pod \"f2d4e516-45b8-4f83-9610-589825ea5d90\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") "
Jul 09 16:48:04.594324 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594252    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd6sg\" (UniqueName: \"kubernetes.io/projected/f2d4e516-45b8-4f83-9610-589825ea5d90-kube-api-access-fd6sg\") pod \"f2d4e516-45b8-4f83-9610-589825ea5d90\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") "
Jul 09 16:48:04.594380 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594348    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-model-cache\") pod \"f2d4e516-45b8-4f83-9610-589825ea5d90\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") "
Jul 09 16:48:04.594437 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594371    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f2d4e516-45b8-4f83-9610-589825ea5d90" (UID: "f2d4e516-45b8-4f83-9610-589825ea5d90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:04.594437 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594391    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-dshm\") pod \"f2d4e516-45b8-4f83-9610-589825ea5d90\" (UID: \"f2d4e516-45b8-4f83-9610-589825ea5d90\") "
Jul 09 16:48:04.594556 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594530    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-home" (OuterVolumeSpecName: "home") pod "f2d4e516-45b8-4f83-9610-589825ea5d90" (UID: "f2d4e516-45b8-4f83-9610-589825ea5d90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:04.594721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594689    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-model-cache" (OuterVolumeSpecName: "model-cache") pod "f2d4e516-45b8-4f83-9610-589825ea5d90" (UID: "f2d4e516-45b8-4f83-9610-589825ea5d90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:04.594721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594716    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:04.594862 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.594735    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:04.595500 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.595482    2558 scope.go:117] "RemoveContainer" containerID="8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718"
Jul 09 16:48:04.595820 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:04.595797    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718\": container with ID starting with 8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718 not found: ID does not exist" containerID="8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718"
Jul 09 16:48:04.595938 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.595832    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718"} err="failed to get container status \"8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718\": rpc error: code = NotFound desc = could not find container \"8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718\": container with ID starting with 8dc227557cff7af3553311063ee7303cf287f04b6c3db0cf37230a84de0d3718 not found: ID does not exist"
Jul 09 16:48:04.596442 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.596414    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2d4e516-45b8-4f83-9610-589825ea5d90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f2d4e516-45b8-4f83-9610-589825ea5d90" (UID: "f2d4e516-45b8-4f83-9610-589825ea5d90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:48:04.596532 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.596507    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-dshm" (OuterVolumeSpecName: "dshm") pod "f2d4e516-45b8-4f83-9610-589825ea5d90" (UID: "f2d4e516-45b8-4f83-9610-589825ea5d90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:04.596665 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.596645    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2d4e516-45b8-4f83-9610-589825ea5d90-kube-api-access-fd6sg" (OuterVolumeSpecName: "kube-api-access-fd6sg") pod "f2d4e516-45b8-4f83-9610-589825ea5d90" (UID: "f2d4e516-45b8-4f83-9610-589825ea5d90"). InnerVolumeSpecName "kube-api-access-fd6sg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:04.696006 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.695967    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2d4e516-45b8-4f83-9610-589825ea5d90-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:04.696006 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.696000    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd6sg\" (UniqueName: \"kubernetes.io/projected/f2d4e516-45b8-4f83-9610-589825ea5d90-kube-api-access-fd6sg\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:04.696006 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.696010    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:04.696256 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.696020    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2d4e516-45b8-4f83-9610-589825ea5d90-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:04.909084 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.909048    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"]
Jul 09 16:48:04.915778 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:04.915742    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-a36bf9/rate-test-model-a36bf9-kserve-6bb8bc8f99-4bjpn"]
Jul 09 16:48:05.173710 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.173580    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx" podUID="8a615bc0-2be3-4ab6-92d6-1c60953a7792" containerName="main" containerID="cri-o://7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc" gracePeriod=45
Jul 09 16:48:05.179583 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.179549    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2d4e516-45b8-4f83-9610-589825ea5d90" path="/var/lib/kubelet/pods/f2d4e516-45b8-4f83-9610-589825ea5d90/volumes"
Jul 09 16:48:05.426399 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.426337    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:48:05.502441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502401    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69cxs\" (UniqueName: \"kubernetes.io/projected/8a615bc0-2be3-4ab6-92d6-1c60953a7792-kube-api-access-69cxs\") pod \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") "
Jul 09 16:48:05.502622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502461    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tls-certs\") pod \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") "
Jul 09 16:48:05.502622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502537    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-home\") pod \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") "
Jul 09 16:48:05.502622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502572    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-model-cache\") pod \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") "
Jul 09 16:48:05.502622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502604    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tmp-dir\") pod \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") "
Jul 09 16:48:05.502622 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502621    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-dshm\") pod \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\" (UID: \"8a615bc0-2be3-4ab6-92d6-1c60953a7792\") "
Jul 09 16:48:05.502945 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502920    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a615bc0-2be3-4ab6-92d6-1c60953a7792" (UID: "8a615bc0-2be3-4ab6-92d6-1c60953a7792"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:05.503005 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502944    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a615bc0-2be3-4ab6-92d6-1c60953a7792" (UID: "8a615bc0-2be3-4ab6-92d6-1c60953a7792"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:05.503005 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.502920    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-home" (OuterVolumeSpecName: "home") pod "8a615bc0-2be3-4ab6-92d6-1c60953a7792" (UID: "8a615bc0-2be3-4ab6-92d6-1c60953a7792"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:05.504847 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.504821    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a615bc0-2be3-4ab6-92d6-1c60953a7792" (UID: "8a615bc0-2be3-4ab6-92d6-1c60953a7792"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:48:05.504847 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.504835    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a615bc0-2be3-4ab6-92d6-1c60953a7792-kube-api-access-69cxs" (OuterVolumeSpecName: "kube-api-access-69cxs") pod "8a615bc0-2be3-4ab6-92d6-1c60953a7792" (UID: "8a615bc0-2be3-4ab6-92d6-1c60953a7792"). InnerVolumeSpecName "kube-api-access-69cxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:05.505021 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.504823    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-dshm" (OuterVolumeSpecName: "dshm") pod "8a615bc0-2be3-4ab6-92d6-1c60953a7792" (UID: "8a615bc0-2be3-4ab6-92d6-1c60953a7792"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:05.593473 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.593434    2558 generic.go:356] "Generic (PLEG): container finished" podID="8a615bc0-2be3-4ab6-92d6-1c60953a7792" containerID="7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc" exitCode=0
Jul 09 16:48:05.593473 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.593476    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"
Jul 09 16:48:05.593704 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.593516    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx" event={"ID":"8a615bc0-2be3-4ab6-92d6-1c60953a7792","Type":"ContainerDied","Data":"7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc"}
Jul 09 16:48:05.593704 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.593560    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx" event={"ID":"8a615bc0-2be3-4ab6-92d6-1c60953a7792","Type":"ContainerDied","Data":"67ac2cdb4ecfd9b25c7fcedb119392e34cb1630894ce25ee5cced344f9efbf33"}
Jul 09 16:48:05.593704 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.593581    2558 scope.go:117] "RemoveContainer" containerID="7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc"
Jul 09 16:48:05.603036 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603015    2558 scope.go:117] "RemoveContainer" containerID="7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc"
Jul 09 16:48:05.603284 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:05.603265    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc\": container with ID starting with 7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc not found: ID does not exist" containerID="7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc"
Jul 09 16:48:05.603350 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603293    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc"} err="failed to get container status \"7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc\": rpc error: code = NotFound desc = could not find container \"7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc\": container with ID starting with 7ea6af1121e416d156194f6ec86dbeda19e7ae852f0f3fbc2f96abd2165800cc not found: ID does not exist"
Jul 09 16:48:05.603420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603404    2558 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tmp-dir\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:05.603456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603426    2558 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-dshm\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:05.603456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603435    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cxs\" (UniqueName: \"kubernetes.io/projected/8a615bc0-2be3-4ab6-92d6-1c60953a7792-kube-api-access-69cxs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:05.603456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603443    2558 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a615bc0-2be3-4ab6-92d6-1c60953a7792-tls-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:05.603456 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603452    2558 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-home\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:05.603618 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.603459    2558 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a615bc0-2be3-4ab6-92d6-1c60953a7792-model-cache\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:05.615087 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.615064    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"]
Jul 09 16:48:05.621026 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:05.621002    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-fb02fd/rate-test-model-fb02fd-kserve-77fb594dc9-nvhsx"]
Jul 09 16:48:07.178309 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:07.178271    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a615bc0-2be3-4ab6-92d6-1c60953a7792" path="/var/lib/kubelet/pods/8a615bc0-2be3-4ab6-92d6-1c60953a7792/volumes"
Jul 09 16:48:09.638505 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:09.638474    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"]
Jul 09 16:48:09.639003 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:09.638759    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" podUID="bcac3a2b-b97e-4398-b031-71e5e102d219" containerName="maas-api" containerID="cri-o://763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9" gracePeriod=30
Jul 09 16:48:10.085192 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.085168    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:48:10.145212 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.145177    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznbl\" (UniqueName: \"kubernetes.io/projected/bcac3a2b-b97e-4398-b031-71e5e102d219-kube-api-access-hznbl\") pod \"bcac3a2b-b97e-4398-b031-71e5e102d219\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") "
Jul 09 16:48:10.145212 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.145216    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls\") pod \"bcac3a2b-b97e-4398-b031-71e5e102d219\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") "
Jul 09 16:48:10.145412 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.145339    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcac3a2b-b97e-4398-b031-71e5e102d219-openshift-service-ca\") pod \"bcac3a2b-b97e-4398-b031-71e5e102d219\" (UID: \"bcac3a2b-b97e-4398-b031-71e5e102d219\") "
Jul 09 16:48:10.145691 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.145667    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcac3a2b-b97e-4398-b031-71e5e102d219-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcac3a2b-b97e-4398-b031-71e5e102d219" (UID: "bcac3a2b-b97e-4398-b031-71e5e102d219"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:48:10.147459 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.147441    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcac3a2b-b97e-4398-b031-71e5e102d219" (UID: "bcac3a2b-b97e-4398-b031-71e5e102d219"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:48:10.147536 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.147520    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcac3a2b-b97e-4398-b031-71e5e102d219-kube-api-access-hznbl" (OuterVolumeSpecName: "kube-api-access-hznbl") pod "bcac3a2b-b97e-4398-b031-71e5e102d219" (UID: "bcac3a2b-b97e-4398-b031-71e5e102d219"). InnerVolumeSpecName "kube-api-access-hznbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:10.246597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.246504    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcac3a2b-b97e-4398-b031-71e5e102d219-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:10.246597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.246535    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznbl\" (UniqueName: \"kubernetes.io/projected/bcac3a2b-b97e-4398-b031-71e5e102d219-kube-api-access-hznbl\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:10.246597 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.246544    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcac3a2b-b97e-4398-b031-71e5e102d219-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:10.621663 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.621629    2558 generic.go:356] "Generic (PLEG): container finished" podID="bcac3a2b-b97e-4398-b031-71e5e102d219" containerID="763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9" exitCode=0
Jul 09 16:48:10.621855 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.621680    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" event={"ID":"bcac3a2b-b97e-4398-b031-71e5e102d219","Type":"ContainerDied","Data":"763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9"}
Jul 09 16:48:10.621855 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.621684    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"
Jul 09 16:48:10.621855 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.621722    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw" event={"ID":"bcac3a2b-b97e-4398-b031-71e5e102d219","Type":"ContainerDied","Data":"97a09c36119ed408744e46646744fb6fe4011724a16c5b9eed251e0c68fd12f8"}
Jul 09 16:48:10.621855 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.621745    2558 scope.go:117] "RemoveContainer" containerID="763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9"
Jul 09 16:48:10.632257 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.632241    2558 scope.go:117] "RemoveContainer" containerID="763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9"
Jul 09 16:48:10.632509 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:10.632490    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9\": container with ID starting with 763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9 not found: ID does not exist" containerID="763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9"
Jul 09 16:48:10.632564 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.632516    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9"} err="failed to get container status \"763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9\": rpc error: code = NotFound desc = could not find container \"763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9\": container with ID starting with 763f18bd04f04c4639f32003ce2cae92cf93afc4becad32ff8f133d6a05254f9 not found: ID does not exist"
Jul 09 16:48:10.645257 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.645234    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"]
Jul 09 16:48:10.648502 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:10.648482    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-a36bf9-d58c959b4-dqctw"]
Jul 09 16:48:11.178658 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:11.178624    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcac3a2b-b97e-4398-b031-71e5e102d219" path="/var/lib/kubelet/pods/bcac3a2b-b97e-4398-b031-71e5e102d219/volumes"
Jul 09 16:48:15.760565 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:15.760473    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"]
Jul 09 16:48:15.760975 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:15.760758    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" containerID="cri-o://b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5" gracePeriod=30
Jul 09 16:48:16.983179 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:16.983145    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"]
Jul 09 16:48:16.983686 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:16.983388    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" podUID="83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" containerName="maas-api" containerID="cri-o://d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290" gracePeriod=30
Jul 09 16:48:17.444982 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.444958    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:48:17.518505 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.518420    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls\") pod \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") "
Jul 09 16:48:17.518505 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.518462    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46t7p\" (UniqueName: \"kubernetes.io/projected/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-kube-api-access-46t7p\") pod \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") "
Jul 09 16:48:17.518728 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.518576    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-openshift-service-ca\") pod \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\" (UID: \"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e\") "
Jul 09 16:48:17.519080 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.519049    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" (UID: "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:48:17.520695 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.520673    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" (UID: "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:48:17.520798 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.520782    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-kube-api-access-46t7p" (OuterVolumeSpecName: "kube-api-access-46t7p") pod "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" (UID: "83f50dd2-0d3b-4be7-8b30-cf771dd3de4e"). InnerVolumeSpecName "kube-api-access-46t7p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:17.619971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.619929    2558 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-openshift-service-ca\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:17.619971 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.619967    2558 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-maas-api-tls\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:17.620214 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.619984    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46t7p\" (UniqueName: \"kubernetes.io/projected/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e-kube-api-access-46t7p\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:17.660932 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.660871    2558 generic.go:356] "Generic (PLEG): container finished" podID="83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" containerID="d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290" exitCode=0
Jul 09 16:48:17.661066 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.660939    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"
Jul 09 16:48:17.661066 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.660985    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" event={"ID":"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e","Type":"ContainerDied","Data":"d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290"}
Jul 09 16:48:17.661138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.661040    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7" event={"ID":"83f50dd2-0d3b-4be7-8b30-cf771dd3de4e","Type":"ContainerDied","Data":"ed5ee8b5c379a214a6564dac9a516fe50417c3eb21042c5d3befe0b811c896bf"}
Jul 09 16:48:17.661138 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.661091    2558 scope.go:117] "RemoveContainer" containerID="d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290"
Jul 09 16:48:17.670619 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.670599    2558 scope.go:117] "RemoveContainer" containerID="d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290"
Jul 09 16:48:17.670861 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:17.670840    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290\": container with ID starting with d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290 not found: ID does not exist" containerID="d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290"
Jul 09 16:48:17.670997 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.670873    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290"} err="failed to get container status \"d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290\": rpc error: code = NotFound desc = could not find container \"d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290\": container with ID starting with d3212f6ceb22d34dc19d13bfee303733ecaabde98c6b976ae13acf8a7734d290 not found: ID does not exist"
Jul 09 16:48:17.683077 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.683048    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"]
Jul 09 16:48:17.685548 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:17.685529    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-fb02fd-75dc8cb8c4-b9fw7"]
Jul 09 16:48:19.178098 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:19.178064    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" path="/var/lib/kubelet/pods/83f50dd2-0d3b-4be7-8b30-cf771dd3de4e/volumes"
Jul 09 16:48:19.509870 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:19.509771    2558 patch_prober.go:28] interesting pod/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 16:48:19.510052 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:19.509860    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:48:21.022826 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.022753    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:48:21.152574 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152523    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-data\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.152782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152625    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-credential-socket\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.152782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152654    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-socket\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.152782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152677    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-822nd\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-kube-api-access-822nd\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.152782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152727    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-certs\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.152782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152748    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-envoy\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.152782 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152778    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-podinfo\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.153170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152812    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b932c0-f451-4cee-9f12-31a686398ef4-istiod-ca-cert\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.153170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152851    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-token\") pod \"a6b932c0-f451-4cee-9f12-31a686398ef4\" (UID: \"a6b932c0-f451-4cee-9f12-31a686398ef4\") "
Jul 09 16:48:21.153170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.152980    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:21.153170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.153103    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:21.153393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.153288    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.153393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.153307    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.153393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.153332    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:21.153393 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.153375    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b932c0-f451-4cee-9f12-31a686398ef4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:48:21.153541 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.153469    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-data" (OuterVolumeSpecName: "istio-data") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:21.155198 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.155169    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:48:21.155322 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.155209    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:21.155322 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.155236    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-kube-api-access-822nd" (OuterVolumeSpecName: "kube-api-access-822nd") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "kube-api-access-822nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:21.155490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.155472    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-token" (OuterVolumeSpecName: "istio-token") pod "a6b932c0-f451-4cee-9f12-31a686398ef4" (UID: "a6b932c0-f451-4cee-9f12-31a686398ef4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:21.254352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254305    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.254352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254348    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.254352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254361    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-822nd\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-kube-api-access-822nd\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.254656 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254372    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.254656 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254381    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.254656 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254388    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6b932c0-f451-4cee-9f12-31a686398ef4-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.254656 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.254396    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6b932c0-f451-4cee-9f12-31a686398ef4-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:21.685630 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.685600    2558 generic.go:356] "Generic (PLEG): container finished" podID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerID="b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5" exitCode=0
Jul 09 16:48:21.685806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.685653    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"
Jul 09 16:48:21.685806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.685672    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" event={"ID":"a6b932c0-f451-4cee-9f12-31a686398ef4","Type":"ContainerDied","Data":"b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5"}
Jul 09 16:48:21.685806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.685715    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4" event={"ID":"a6b932c0-f451-4cee-9f12-31a686398ef4","Type":"ContainerDied","Data":"e7e6878df6b0648bc5c7b189ee4da1d4e0b0dfd1b89bcaa47a64abc26f39e462"}
Jul 09 16:48:21.685806 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.685736    2558 scope.go:117] "RemoveContainer" containerID="b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5"
Jul 09 16:48:21.695043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.695023    2558 scope.go:117] "RemoveContainer" containerID="743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"
Jul 09 16:48:21.703345 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.703326    2558 scope.go:117] "RemoveContainer" containerID="b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5"
Jul 09 16:48:21.703713 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:21.703690    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5\": container with ID starting with b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5 not found: ID does not exist" containerID="b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5"
Jul 09 16:48:21.703814 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.703720    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5"} err="failed to get container status \"b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5\": rpc error: code = NotFound desc = could not find container \"b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5\": container with ID starting with b674dfa780f8762a6b0689342059ea158fdbaf90933432318546c44c1dc7e7b5 not found: ID does not exist"
Jul 09 16:48:21.703814 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.703740    2558 scope.go:117] "RemoveContainer" containerID="743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"
Jul 09 16:48:21.704162 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:21.704140    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1\": container with ID starting with 743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1 not found: ID does not exist" containerID="743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"
Jul 09 16:48:21.704263 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.704167    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1"} err="failed to get container status \"743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1\": rpc error: code = NotFound desc = could not find container \"743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1\": container with ID starting with 743234e2a9b67fea590f194895f519458fc3e16a61ee246b7110f19b6f2841b1 not found: ID does not exist"
Jul 09 16:48:21.707912 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.707828    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"]
Jul 09 16:48:21.710286 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:21.710260    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-a36bf9-openshift-default-8559b7d459-54ln4"]
Jul 09 16:48:22.808929 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:22.808869    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"]
Jul 09 16:48:22.809342 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:22.809155    2558 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" containerID="cri-o://e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb" gracePeriod=30
Jul 09 16:48:23.178048 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:23.178009    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" path="/var/lib/kubelet/pods/a6b932c0-f451-4cee-9f12-31a686398ef4/volumes"
Jul 09 16:48:23.488818 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:23.488733    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d6bc4c8c4-6q656_5d330e9e-97ce-44bf-b028-f0d4eec96434/authorino/0.log"
Jul 09 16:48:26.983850 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:26.983808    2558 patch_prober.go:28] interesting pod/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 16:48:26.984253 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:26.983866    2558 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:48:28.064096 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.064072    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:48:28.114865 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.114784    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd244e0-2708-459c-9fdd-42093e7661bd-istiod-ca-cert\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.114865 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.114856    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-credential-socket\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.114907    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-certs\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.114958    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-envoy\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.114982    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-token\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115017    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-podinfo\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115041    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4bxsd\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-kube-api-access-4bxsd\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115116 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115071    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-socket\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115422 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115127    2558 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-data\") pod \"ebd244e0-2708-459c-9fdd-42093e7661bd\" (UID: \"ebd244e0-2708-459c-9fdd-42093e7661bd\") "
Jul 09 16:48:28.115422 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115199    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd244e0-2708-459c-9fdd-42093e7661bd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:48:28.115422 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115334    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:28.115578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115451    2558 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebd244e0-2708-459c-9fdd-42093e7661bd-istiod-ca-cert\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.115578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115469    2558 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-credential-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.115578 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115553    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:28.115738 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.115687    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:28.116411 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.116388    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:28.117542 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.117517    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:28.117672 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.117599    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:48:28.117825 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.117809    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-kube-api-access-4bxsd" (OuterVolumeSpecName: "kube-api-access-4bxsd") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "kube-api-access-4bxsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:48:28.117928 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.117908    2558 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebd244e0-2708-459c-9fdd-42093e7661bd" (UID: "ebd244e0-2708-459c-9fdd-42093e7661bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 16:48:28.216372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216340    2558 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-certs\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.216372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216363    2558 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-envoy\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.216372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216372    2558 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-token\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.216372 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216379    2558 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-podinfo\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.216720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216387    2558 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxsd\" (UniqueName: \"kubernetes.io/projected/ebd244e0-2708-459c-9fdd-42093e7661bd-kube-api-access-4bxsd\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.216720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216397    2558 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-workload-socket\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.216720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.216404    2558 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebd244e0-2708-459c-9fdd-42093e7661bd-istio-data\") on node \"ip-10-0-133-91.ec2.internal\" DevicePath \"\""
Jul 09 16:48:28.355323 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.355292    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpfzl_0fec7e61-b970-440e-aa5c-0bfcf20f8b3e/manager/0.log"
Jul 09 16:48:28.469581 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.469499    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b9sf7_da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8/manager/0.log"
Jul 09 16:48:28.726150 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.726050    2558 generic.go:356] "Generic (PLEG): container finished" podID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerID="e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb" exitCode=0
Jul 09 16:48:28.726150 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.726105    2558 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"
Jul 09 16:48:28.726150 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.726110    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" event={"ID":"ebd244e0-2708-459c-9fdd-42093e7661bd","Type":"ContainerDied","Data":"e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb"}
Jul 09 16:48:28.726420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.726166    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk" event={"ID":"ebd244e0-2708-459c-9fdd-42093e7661bd","Type":"ContainerDied","Data":"bbb988a56b592d3f4deda89c25206a0af6d5757e4cd80fdd8eff10779a891a2e"}
Jul 09 16:48:28.726420 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.726196    2558 scope.go:117] "RemoveContainer" containerID="e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb"
Jul 09 16:48:28.735906 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.735865    2558 scope.go:117] "RemoveContainer" containerID="38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"
Jul 09 16:48:28.744741 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.744717    2558 scope.go:117] "RemoveContainer" containerID="e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb"
Jul 09 16:48:28.745073 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:28.745053    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb\": container with ID starting with e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb not found: ID does not exist" containerID="e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb"
Jul 09 16:48:28.745123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.745083    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb"} err="failed to get container status \"e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb\": rpc error: code = NotFound desc = could not find container \"e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb\": container with ID starting with e3317448d4ce600517ab8e7c0755f4d877bf0ceefc96de0b6d3270d2f2c2b6eb not found: ID does not exist"
Jul 09 16:48:28.745123 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.745105    2558 scope.go:117] "RemoveContainer" containerID="38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"
Jul 09 16:48:28.745372 ip-10-0-133-91 kubenswrapper[2558]: E0709 16:48:28.745354    2558 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d\": container with ID starting with 38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d not found: ID does not exist" containerID="38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"
Jul 09 16:48:28.745425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.745380    2558 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d"} err="failed to get container status \"38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d\": rpc error: code = NotFound desc = could not find container \"38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d\": container with ID starting with 38d485c17ec4f6e3d84dc480a4bb200c28b1b56552ad48e772cba68ef240748d not found: ID does not exist"
Jul 09 16:48:28.748726 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.748694    2558 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"]
Jul 09 16:48:28.757101 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.757071    2558 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-fb02fd-openshift-default-59699b555f-4ghvk"]
Jul 09 16:48:28.821764 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:28.821733    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c5466_4d2bc1d3-113b-4e76-94a9-a162c915e7d0/manager/0.log"
Jul 09 16:48:29.061767 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:29.061688    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757d6df76f-cgmf8_3fb1ebae-a1a3-468f-ada7-4f5ca58b7901/copy-manifests/0.log"
Jul 09 16:48:29.073858 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:29.073822    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757d6df76f-cgmf8_3fb1ebae-a1a3-468f-ada7-4f5ca58b7901/manager/0.log"
Jul 09 16:48:29.177739 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:29.177706    2558 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" path="/var/lib/kubelet/pods/ebd244e0-2708-459c-9fdd-42093e7661bd/volumes"
Jul 09 16:48:30.381132 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.381100    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv_c34ab2f4-8edc-4723-8578-3edfdb0e7169/util/0.log"
Jul 09 16:48:30.386990 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.386957    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv_c34ab2f4-8edc-4723-8578-3edfdb0e7169/pull/0.log"
Jul 09 16:48:30.392574 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.392553    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv_c34ab2f4-8edc-4723-8578-3edfdb0e7169/extract/0.log"
Jul 09 16:48:30.514735 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.514703    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4_418616e9-35f0-4127-b3f3-9c3d582ff28c/util/0.log"
Jul 09 16:48:30.520916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.520896    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4_418616e9-35f0-4127-b3f3-9c3d582ff28c/pull/0.log"
Jul 09 16:48:30.527043 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.527019    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4_418616e9-35f0-4127-b3f3-9c3d582ff28c/extract/0.log"
Jul 09 16:48:30.638703 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.638615    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh_3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b/extract/0.log"
Jul 09 16:48:30.644496 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.644472    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh_3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b/util/0.log"
Jul 09 16:48:30.650475 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.650460    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh_3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b/pull/0.log"
Jul 09 16:48:30.766659 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.766636    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9_21ff865d-6fc8-4ec9-a663-5a6728b6c993/util/0.log"
Jul 09 16:48:30.772774 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.772754    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9_21ff865d-6fc8-4ec9-a663-5a6728b6c993/pull/0.log"
Jul 09 16:48:30.780129 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.780087    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9_21ff865d-6fc8-4ec9-a663-5a6728b6c993/extract/0.log"
Jul 09 16:48:30.899308 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:30.899218    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d6bc4c8c4-6q656_5d330e9e-97ce-44bf-b028-f0d4eec96434/authorino/0.log"
Jul 09 16:48:31.010865 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:31.010834    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-czqxr_413c821b-531a-475f-8dd4-9c7336799a16/manager/0.log"
Jul 09 16:48:31.117606 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:31.117578    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cc6pf_cb03679a-9dd3-4449-b31f-79c34d2dcfdd/manager/0.log"
Jul 09 16:48:31.345830 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:31.345798    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k8xfw_567f9a6c-5db1-4a88-b251-cd767e7bdb4b/registry-server/0.log"
Jul 09 16:48:31.457616 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:31.457579    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9dtbg_f860d58f-fe0e-4199-82a3-3ad1523707e0/manager/0.log"
Jul 09 16:48:31.687365 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:31.687287    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hc5c4_cc8e8027-3a55-4e3a-8454-fc23ecc7abbc/manager/0.log"
Jul 09 16:48:32.029124 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:32.029040    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788znpb_05d99eee-bd0f-41d3-8ef8-3104dc456277/istio-proxy/0.log"
Jul 09 16:48:32.247555 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:32.247524    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-566b87dd6c-65ct9_12c930d5-0791-4b77-9875-eb7dd853ce00/kube-auth-proxy/0.log"
Jul 09 16:48:32.466417 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:32.466392    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/1.log"
Jul 09 16:48:33.144453 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:33.144426    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-t9wsr_702fdce0-61ce-4b26-b14c-c35a3002b250/main/0.log"
Jul 09 16:48:37.588743 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.588706    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gcfwn/must-gather-v6l2t"]
Jul 09 16:48:37.589252 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589232    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589255    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589274    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcac3a2b-b97e-4398-b031-71e5e102d219" containerName="maas-api"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589282    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcac3a2b-b97e-4398-b031-71e5e102d219" containerName="maas-api"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589293    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a615bc0-2be3-4ab6-92d6-1c60953a7792" containerName="main"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589301    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a615bc0-2be3-4ab6-92d6-1c60953a7792" containerName="main"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589313    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" containerName="maas-api"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589320    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" containerName="maas-api"
Jul 09 16:48:37.589333 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589334    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2d4e516-45b8-4f83-9610-589825ea5d90" containerName="main"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589343    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2d4e516-45b8-4f83-9610-589825ea5d90" containerName="main"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589360    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589369    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589379    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589386    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589465    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589475    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a615bc0-2be3-4ab6-92d6-1c60953a7792" containerName="main"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589483    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2d4e516-45b8-4f83-9610-589825ea5d90" containerName="main"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589490    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589497    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcac3a2b-b97e-4398-b031-71e5e102d219" containerName="maas-api"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589506    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="83f50dd2-0d3b-4be7-8b30-cf771dd3de4e" containerName="maas-api"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589517    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589592    2558 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589602    2558 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b932c0-f451-4cee-9f12-31a686398ef4" containerName="istio-proxy"
Jul 09 16:48:37.589721 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.589687    2558 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd244e0-2708-459c-9fdd-42093e7661bd" containerName="istio-proxy"
Jul 09 16:48:37.594095 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.594074    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.596909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.596889    2558 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gcfwn\"/\"default-dockercfg-6d4ng\""
Jul 09 16:48:37.596909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.596898    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gcfwn\"/\"openshift-service-ca.crt\""
Jul 09 16:48:37.597092 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.597067    2558 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gcfwn\"/\"kube-root-ca.crt\""
Jul 09 16:48:37.600972 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.600948    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gcfwn/must-gather-v6l2t"]
Jul 09 16:48:37.690628 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.690596    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-469q6\" (UniqueName: \"kubernetes.io/projected/e022292b-c9a9-4650-98e9-962d3ce16415-kube-api-access-469q6\") pod \"must-gather-v6l2t\" (UID: \"e022292b-c9a9-4650-98e9-962d3ce16415\") " pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.690791 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.690648    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e022292b-c9a9-4650-98e9-962d3ce16415-must-gather-output\") pod \"must-gather-v6l2t\" (UID: \"e022292b-c9a9-4650-98e9-962d3ce16415\") " pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.792075 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.792036    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-469q6\" (UniqueName: \"kubernetes.io/projected/e022292b-c9a9-4650-98e9-962d3ce16415-kube-api-access-469q6\") pod \"must-gather-v6l2t\" (UID: \"e022292b-c9a9-4650-98e9-962d3ce16415\") " pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.792241 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.792106    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e022292b-c9a9-4650-98e9-962d3ce16415-must-gather-output\") pod \"must-gather-v6l2t\" (UID: \"e022292b-c9a9-4650-98e9-962d3ce16415\") " pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.792431 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.792412    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e022292b-c9a9-4650-98e9-962d3ce16415-must-gather-output\") pod \"must-gather-v6l2t\" (UID: \"e022292b-c9a9-4650-98e9-962d3ce16415\") " pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.803445 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.803424    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-469q6\" (UniqueName: \"kubernetes.io/projected/e022292b-c9a9-4650-98e9-962d3ce16415-kube-api-access-469q6\") pod \"must-gather-v6l2t\" (UID: \"e022292b-c9a9-4650-98e9-962d3ce16415\") " pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:37.903698 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:37.903623    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gcfwn/must-gather-v6l2t"
Jul 09 16:48:38.030979 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:38.030951    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gcfwn/must-gather-v6l2t"]
Jul 09 16:48:38.032561 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:48:38.032530    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode022292b_c9a9_4650_98e9_962d3ce16415.slice/crio-c4b503b7b4870e911c4a8b53c19e0b8019466a9ec7f4406718166effa12f8eb8 WatchSource:0}: Error finding container c4b503b7b4870e911c4a8b53c19e0b8019466a9ec7f4406718166effa12f8eb8: Status 404 returned error can't find the container with id c4b503b7b4870e911c4a8b53c19e0b8019466a9ec7f4406718166effa12f8eb8
Jul 09 16:48:38.034483 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:38.034464    2558 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:48:38.778013 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:38.777975    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gcfwn/must-gather-v6l2t" event={"ID":"e022292b-c9a9-4650-98e9-962d3ce16415","Type":"ContainerStarted","Data":"c4b503b7b4870e911c4a8b53c19e0b8019466a9ec7f4406718166effa12f8eb8"}
Jul 09 16:48:39.793349 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:39.792536    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gcfwn/must-gather-v6l2t" event={"ID":"e022292b-c9a9-4650-98e9-962d3ce16415","Type":"ContainerStarted","Data":"d373594960909573f3698677cbc74280d06c02c0474552421af8f8492ed4a26b"}
Jul 09 16:48:39.793349 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:39.792599    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gcfwn/must-gather-v6l2t" event={"ID":"e022292b-c9a9-4650-98e9-962d3ce16415","Type":"ContainerStarted","Data":"a4dce00880647db26a6fda6ed621789336928face289ce61fc02015e802ada5e"}
Jul 09 16:48:39.808490 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:39.808445    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gcfwn/must-gather-v6l2t" podStartSLOduration=1.690641679 podStartE2EDuration="2.808427924s" totalImagesPullingTime="1.117786245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:48:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:48:39.808307877 +0000 UTC m=+2665.190124668" watchObservedRunningTime="2026-07-09 16:48:39.808427924 +0000 UTC m=+2665.190244711"
Jul 09 16:48:40.686651 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:40.686606    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x9k9w_ed33335f-5aec-4360-9937-e478cd9fbe04/global-pull-secret-syncer/0.log"
Jul 09 16:48:40.773457 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:40.773424    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hhw4n_a539fd7d-b1a9-4708-a562-1dfe76c9dc0f/konnectivity-agent/0.log"
Jul 09 16:48:40.825635 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:40.825602    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-91.ec2.internal_0964284bd0ac4fa0d88f69cd97267984/haproxy/0.log"
Jul 09 16:48:45.318916 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.318846    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv_c34ab2f4-8edc-4723-8578-3edfdb0e7169/extract/0.log"
Jul 09 16:48:45.342939 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.342915    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv_c34ab2f4-8edc-4723-8578-3edfdb0e7169/util/0.log"
Jul 09 16:48:45.376296 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.376257    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759lrxcv_c34ab2f4-8edc-4723-8578-3edfdb0e7169/pull/0.log"
Jul 09 16:48:45.406720 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.406686    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4_418616e9-35f0-4127-b3f3-9c3d582ff28c/extract/0.log"
Jul 09 16:48:45.427898 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.427855    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4_418616e9-35f0-4127-b3f3-9c3d582ff28c/util/0.log"
Jul 09 16:48:45.460756 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.460720    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0qhvm4_418616e9-35f0-4127-b3f3-9c3d582ff28c/pull/0.log"
Jul 09 16:48:45.492567 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.492530    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh_3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b/extract/0.log"
Jul 09 16:48:45.513276 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.513251    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh_3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b/util/0.log"
Jul 09 16:48:45.533440 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.533415    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73fkswh_3cb63f40-aeb9-487b-8bb2-e2ba3977ae8b/pull/0.log"
Jul 09 16:48:45.564741 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.564717    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9_21ff865d-6fc8-4ec9-a663-5a6728b6c993/extract/0.log"
Jul 09 16:48:45.585568 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.585484    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9_21ff865d-6fc8-4ec9-a663-5a6728b6c993/util/0.log"
Jul 09 16:48:45.609441 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:45.609408    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vzp9_21ff865d-6fc8-4ec9-a663-5a6728b6c993/pull/0.log"
Jul 09 16:48:46.045946 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:46.045298    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-d6bc4c8c4-6q656_5d330e9e-97ce-44bf-b028-f0d4eec96434/authorino/0.log"
Jul 09 16:48:46.081901 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:46.081850    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-czqxr_413c821b-531a-475f-8dd4-9c7336799a16/manager/0.log"
Jul 09 16:48:46.108110 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:46.108080    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-cc6pf_cb03679a-9dd3-4449-b31f-79c34d2dcfdd/manager/0.log"
Jul 09 16:48:46.177270 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:46.177234    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-k8xfw_567f9a6c-5db1-4a88-b251-cd767e7bdb4b/registry-server/0.log"
Jul 09 16:48:46.309687 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:46.309116    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-9dtbg_f860d58f-fe0e-4199-82a3-3ad1523707e0/manager/0.log"
Jul 09 16:48:46.408231 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:46.408195    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-hc5c4_cc8e8027-3a55-4e3a-8454-fc23ecc7abbc/manager/0.log"
Jul 09 16:48:48.370482 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:48.370448    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6bf65bc649-r2mk2_355658df-4653-4828-be43-cd8b3e8b0135/metrics-server/0.log"
Jul 09 16:48:48.561165 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:48.561133    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jl9hb_4ab8077e-455d-4d43-94ec-b192d34a6d4c/node-exporter/0.log"
Jul 09 16:48:48.580921 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:48.580830    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jl9hb_4ab8077e-455d-4d43-94ec-b192d34a6d4c/kube-rbac-proxy/0.log"
Jul 09 16:48:48.603304 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:48.603279    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jl9hb_4ab8077e-455d-4d43-94ec-b192d34a6d4c/init-textfile/0.log"
Jul 09 16:48:50.688253 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.688221    2558 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"]
Jul 09 16:48:50.693225 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.693191    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.701580 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.701550    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"]
Jul 09 16:48:50.830065 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.830027    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-sys\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.830234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.830123    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbhjd\" (UniqueName: \"kubernetes.io/projected/923feada-8e0d-41b9-9572-ba5e6019cb58-kube-api-access-vbhjd\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.830234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.830190    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-podres\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.830322 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.830305    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-proc\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.830554 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.830527    2558 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-lib-modules\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.931624 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.931572    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-sys\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.931811 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.931654    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbhjd\" (UniqueName: \"kubernetes.io/projected/923feada-8e0d-41b9-9572-ba5e6019cb58-kube-api-access-vbhjd\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.931811 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.931696    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-podres\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.931811 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.931706    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-sys\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.931811 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.931795    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-proc\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.932051 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.931845    2558 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-lib-modules\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.932773 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.932746    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-podres\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.932945 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.932786    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-proc\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.933050 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.932941    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/923feada-8e0d-41b9-9572-ba5e6019cb58-lib-modules\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:50.945373 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:50.945305    2558 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbhjd\" (UniqueName: \"kubernetes.io/projected/923feada-8e0d-41b9-9572-ba5e6019cb58-kube-api-access-vbhjd\") pod \"perf-node-gather-daemonset-rdwnd\" (UID: \"923feada-8e0d-41b9-9572-ba5e6019cb58\") " pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:51.007086 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:51.007049    2558 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:51.188631 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:51.185702    2558 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"]
Jul 09 16:48:51.188707 ip-10-0-133-91 kubenswrapper[2558]: W0709 16:48:51.188677    2558 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod923feada_8e0d_41b9_9572_ba5e6019cb58.slice/crio-7aafb70c53698b5abf81d8bfdedfa4556985117a1d445bc35aa56b3764ff89fa WatchSource:0}: Error finding container 7aafb70c53698b5abf81d8bfdedfa4556985117a1d445bc35aa56b3764ff89fa: Status 404 returned error can't find the container with id 7aafb70c53698b5abf81d8bfdedfa4556985117a1d445bc35aa56b3764ff89fa
Jul 09 16:48:51.911458 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:51.911414    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd" event={"ID":"923feada-8e0d-41b9-9572-ba5e6019cb58","Type":"ContainerStarted","Data":"24afba48ebd3978c707488a0438d24ba995274c58354f6ccded739e857e88949"}
Jul 09 16:48:51.911964 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:51.911487    2558 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd" event={"ID":"923feada-8e0d-41b9-9572-ba5e6019cb58","Type":"ContainerStarted","Data":"7aafb70c53698b5abf81d8bfdedfa4556985117a1d445bc35aa56b3764ff89fa"}
Jul 09 16:48:51.912425 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:51.912398    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:51.932380 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:51.932318    2558 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd" podStartSLOduration=1.932297919 podStartE2EDuration="1.932297919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:48:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:48:51.929085828 +0000 UTC m=+2677.310902615" watchObservedRunningTime="2026-07-09 16:48:51.932297919 +0000 UTC m=+2677.314114705"
Jul 09 16:48:52.866547 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:52.866515    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fhktg_36639cf2-b21a-4d83-b9ba-e07f0412759d/dns/0.log"
Jul 09 16:48:52.885919 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:52.885864    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fhktg_36639cf2-b21a-4d83-b9ba-e07f0412759d/kube-rbac-proxy/0.log"
Jul 09 16:48:53.016234 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:53.016207    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-stdmx_04b0d083-c5d4-4cc0-906d-c373319e1c97/dns-node-resolver/0.log"
Jul 09 16:48:53.509065 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:53.509028    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6674bc895c-zf727_8000b1bb-e41b-4608-a2b4-a560d6fbbd69/registry/0.log"
Jul 09 16:48:53.528424 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:53.528396    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-44htx_bd8a93ba-1327-414e-9755-4c2d0ab8a9d8/node-ca/0.log"
Jul 09 16:48:54.453920 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:54.453861    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788znpb_05d99eee-bd0f-41d3-8ef8-3104dc456277/istio-proxy/0.log"
Jul 09 16:48:54.664694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:54.664660    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-566b87dd6c-65ct9_12c930d5-0791-4b77-9875-eb7dd853ce00/kube-auth-proxy/0.log"
Jul 09 16:48:54.712763 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:54.712666    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/0.log"
Jul 09 16:48:54.749961 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:54.749925    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/1.log"
Jul 09 16:48:55.557170 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:55.557137    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jj4z8_a7add5d7-9318-4518-ac96-eb8d62dfc423/serve-healthcheck-canary/0.log"
Jul 09 16:48:56.033729 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:56.033696    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2x8xz_1e5f1cbd-5b16-4b34-a88c-9de143e9c247/kube-rbac-proxy/0.log"
Jul 09 16:48:56.053628 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:56.053598    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2x8xz_1e5f1cbd-5b16-4b34-a88c-9de143e9c247/exporter/0.log"
Jul 09 16:48:56.074609 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:56.074585    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2x8xz_1e5f1cbd-5b16-4b34-a88c-9de143e9c247/extractor/0.log"
Jul 09 16:48:58.223665 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:58.223637    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-zpfzl_0fec7e61-b970-440e-aa5c-0bfcf20f8b3e/manager/0.log"
Jul 09 16:48:58.400210 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:58.400173    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-b9sf7_da7d7c25-f7f2-4df0-b5d2-d01dcb40ead8/manager/0.log"
Jul 09 16:48:58.653025 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:58.652984    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-c5466_4d2bc1d3-113b-4e76-94a9-a162c915e7d0/manager/0.log"
Jul 09 16:48:58.776352 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:58.776321    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757d6df76f-cgmf8_3fb1ebae-a1a3-468f-ada7-4f5ca58b7901/manager/0.log"
Jul 09 16:48:58.791515 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:58.791490    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-757d6df76f-cgmf8_3fb1ebae-a1a3-468f-ada7-4f5ca58b7901/copy-manifests/0.log"
Jul 09 16:48:58.939030 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:58.938933    2558 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gcfwn/perf-node-gather-daemonset-rdwnd"
Jul 09 16:48:59.964505 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:48:59.964453    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-ps4kj_a205263e-e0d7-44d9-8a63-a03aa53b23d0/openshift-lws-operator/0.log"
Jul 09 16:49:00.496479 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:00.496443    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788znpb_05d99eee-bd0f-41d3-8ef8-3104dc456277/istio-proxy/0.log"
Jul 09 16:49:00.702918 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:00.702813    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-566b87dd6c-65ct9_12c930d5-0791-4b77-9875-eb7dd853ce00/kube-auth-proxy/0.log"
Jul 09 16:49:00.747559 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:00.747525    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/0.log"
Jul 09 16:49:00.787588 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:00.787550    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/1.log"
Jul 09 16:49:02.268925 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:02.268816    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb6788znpb_05d99eee-bd0f-41d3-8ef8-3104dc456277/istio-proxy/0.log"
Jul 09 16:49:02.463963 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:02.463926    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-566b87dd6c-65ct9_12c930d5-0791-4b77-9875-eb7dd853ce00/kube-auth-proxy/0.log"
Jul 09 16:49:02.498523 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:02.498496    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/0.log"
Jul 09 16:49:02.537031 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:02.536945    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-nls8r_7851200e-a8e4-4934-8d53-0d483d116ec2/istio-proxy/1.log"
Jul 09 16:49:06.335961 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.335863    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/kube-multus-additional-cni-plugins/0.log"
Jul 09 16:49:06.354694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.354668    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/egress-router-binary-copy/0.log"
Jul 09 16:49:06.372034 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.372014    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/cni-plugins/0.log"
Jul 09 16:49:06.389909 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.389862    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/bond-cni-plugin/0.log"
Jul 09 16:49:06.407793 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.407762    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/routeoverride-cni/0.log"
Jul 09 16:49:06.425666 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.425646    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/whereabouts-cni-bincopy/0.log"
Jul 09 16:49:06.442402 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.442379    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2gqn8_bc73d4b4-53e0-4357-9cc9-1a7c0697629d/whereabouts-cni/0.log"
Jul 09 16:49:06.780083 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.780012    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-m5g2d_6dc1043f-8908-4897-aaa0-a94fc42d5ef5/kube-multus/0.log"
Jul 09 16:49:06.826604 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.826565    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6wzgz_0cca3ed8-8427-4ca7-a1af-7541a27782ff/network-metrics-daemon/0.log"
Jul 09 16:49:06.844723 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:06.844695    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-6wzgz_0cca3ed8-8427-4ca7-a1af-7541a27782ff/kube-rbac-proxy/0.log"
Jul 09 16:49:08.093467 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.093432    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-controller/0.log"
Jul 09 16:49:08.110694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.110649    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/1.log"
Jul 09 16:49:08.110694 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.110675    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovn-acl-logging/0.log"
Jul 09 16:49:08.131314 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.131289    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/kube-rbac-proxy-node/0.log"
Jul 09 16:49:08.150232 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.150203    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/kube-rbac-proxy-ovn-metrics/0.log"
Jul 09 16:49:08.165471 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.165440    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/northd/0.log"
Jul 09 16:49:08.184568 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.184532    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/nbdb/0.log"
Jul 09 16:49:08.202706 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.202680    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/sbdb/0.log"
Jul 09 16:49:08.260209 ip-10-0-133-91 kubenswrapper[2558]: I0709 16:49:08.260182    2558 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-px6rt_d30497fb-b5e7-45da-af81-0ff51251b144/ovnkube-controller/0.log"